--- name: multimodal-radiology-pathology-fusion-agent description: "---name: multimodal-radpath-fusion-agent description: AI-powered multimodal diagnostic fusion integrating radiology imaging (CT/MRI/PET), digital pathology (WSI), genomics, and clinical data for comprehensive cancer diagnosis and treatment planning. license: MIT metadata: author: AI Group version..." --- --- name: 'multimodal-radpath-fusion-agent' description: 'AI-powered multimodal diagnostic fusion integrating radiology imaging (CT/MRI/PET), digital pathology (WSI), genomics, and clinical data for comprehensive cancer diagnosis and treatment planning.' measurable_outcome: Execute skill workflow successfully with valid output within 15 minutes. allowed-tools: - read_file - run_shell_command --- # Multimodal Radiology-Pathology Fusion Agent The **Multimodal Radpath Fusion Agent** integrates diverse clinical data sources including radiology imaging (CT, MRI, PET), digital pathology whole slide images, genomic profiling, and electronic health records using state-of-the-art multimodal deep learning for comprehensive cancer diagnosis, treatment response prediction, and prognostic modeling. ## When to Use This Skill * When integrating radiology and pathology for unified tumor assessment. * For treatment response prediction using multimodal imaging. * To predict molecular features from imaging (imaging genomics). * When building comprehensive prognostic models. * For tumor board decision support with AI second opinion. ## Core Capabilities 1. **Radiology-Pathology Fusion**: Integrate macro and microscopic views. 2. **Imaging-Genomics Correlation**: Predict molecular features from imaging. 3. **Treatment Response Prediction**: Multi-modal response modeling. 4. **Survival Prediction**: Comprehensive prognostic models. 5. **Tumor Characterization**: Integrate phenotype from all modalities. 6. **Clinical Decision Support**: AI-assisted tumor board recommendations. 7. **Dermatology and Dermoscopic Image Triage**: Evaluate clinical and dermoscopic images for basal cell carcinoma and common mimickers as multimodal LLM decision support, with explicit image-quality constraints, differential diagnosis limits, and clinician-overread safeguards. ## Supported Modalities | Modality | Data Type | Features Extracted | |----------|-----------|-------------------| | CT | DICOM volumes | Radiomics, deep features | | MRI | Multi-sequence DICOM | Texture, perfusion, ADC | | PET | SUV maps | Metabolic features | | H&E WSI | SVS/NDPI images | Histology, spatial patterns | | IHC | Stained slides | Biomarker quantification | | WES/WGS | VCF | Mutations, TMB, signatures | | RNA-seq | Expression matrix | Pathway signatures | | Clinical | EHR data | Demographics, labs, history | ## Fusion Architectures | Architecture | Method | Best For | |--------------|--------|----------| | AMRI-Net | Attention fusion | Radiology focus | | PathOmCLIP | Contrastive learning | Path-omics alignment | | SMuRF | Swin Transformer | Multi-region integration | | MultiModal Transformer | Self-attention | All modalities | | GNN Fusion | Graph networks | Spatial relationships | ## Workflow 1. **Data Ingestion**: Collect radiology, pathology, genomics, clinical. 2. **Preprocessing**: Standardize each modality. 3. **Feature Extraction**: Extract modality-specific features. 4. **Alignment**: Temporal and spatial alignment of data. 5. **Fusion**: Multi-modal deep learning integration. 6. **Prediction**: Diagnosis, response, survival prediction. 7. **Output**: Integrated report with explanations. ## Example Usage **User**: "Integrate this lung cancer patient's CT scan, biopsy pathology, and genomic profiling for comprehensive assessment and treatment recommendation." **Agent Action**: ```bash python3 Skills/Clinical/Multimodal_Radpath_Fusion_Agent/multimodal_fusion.py \ --ct_dicom ct_chest/ \ --pet_dicom pet_scan/ \ --wsi_path biopsy.svs \ --genomic_vcf tumor_wes.vcf \ --rna_expression expression.tsv \ --clinical_ehr patient_data.json \ --task treatment_recommendation \ --cancer_type nsclc \ --output integrated_assessment/ ``` ## Output Components | Output | Description | Format | |--------|-------------|--------| | Integrated Diagnosis | Multi-modal classification | .json | | Treatment Prediction | Response probabilities | .json | | Survival Estimate | Prognostic curves | .json, .png | | Feature Attribution | Modality importance | .json | | Attention Maps | Visual explanations | .npy, .png | | Clinical Report | Summary for tumor board | .pdf | | Confidence Scores | Prediction uncertainty | .json | ## Clinical Applications | Application | Modalities | Performance | |-------------|------------|-------------| | NSCLC IO Response | CT + H&E + PD-L1 | AUC 0.85 | | HCC Treatment Selection | MRI + H&E + AFP | AUC 0.82 | | Breast Neoadjuvant | MRI + H&E + HER2 | AUC 0.88 | | HNSCC HPV/Prognosis | CT + H&E + p16 | AUC 0.89 | | GBM Survival | MRI + H&E + MGMT | C-index 0.76 | ## Imaging-Genomics Predictions | Molecular Feature | Imaging Modality | Accuracy | |-------------------|------------------|----------| | EGFR mutation | CT | 75-80% | | KRAS mutation | CT | 70-75% | | PD-L1 expression | CT + H&E | 80-85% | | MSI status | H&E | 85-90% | | TMB level | H&E | 75-80% | | HRD status | H&E | 78-83% | ## AI/ML Components **Feature Extraction**: - 3D ResNet for CT/MRI volumes - Vision Transformers for WSI - Foundation models (CONCH, UNI) **Fusion Methods**: - Cross-attention mechanisms - Multimodal transformers - Contrastive multimodal learning **Prediction Models**: - Multi-task learning - Survival analysis (DeepSurv) - Uncertainty quantification ## Prerequisites * Python 3.10+ * PyTorch, transformers * SimpleITK, OpenSlide * Foundation model weights * GPU with 32GB+ VRAM (recommended) ## Related Skills * Radiomics_Pathomics_Fusion_Agent - Imaging-specific fusion * Pathology_AI/CONCH_Agent - Pathology foundation model * Pan_Cancer_MultiOmics_Agent - Genomic integration * Virtual_Lab_Agent - AI research coordination ## Integration with Clinical Workflow | Integration Point | System | Purpose | |-------------------|--------|---------| | PACS | Radiology archive | Image retrieval | | LIS | Pathology system | Slide access | | EHR | Medical records | Clinical data | | Tumor Board | MDT platform | Decision support | | Reporting | Clinical reports | Documentation | ## Special Considerations 1. **Data Alignment**: Ensure temporal correspondence 2. **Missing Modalities**: Handle incomplete multimodal data 3. **Privacy**: HIPAA compliance for clinical integration 4. **Validation**: Multi-site validation essential 5. **Explainability**: Clinical trust requires interpretability ## Explainability Methods | Method | Output | Purpose | |--------|--------|---------| | Attention Maps | Heatmaps | Important regions | | SHAP Values | Feature importance | Modality contribution | | GradCAM | Activation maps | Visual explanation | | Counterfactuals | What-if analysis | Decision boundaries | ## Quality Control | QC Check | Threshold | Action | |----------|-----------|--------| | Image Quality | Score >0.7 | Flag for review | | Data Completeness | >80% fields | Proceed or wait | | Prediction Confidence | >0.6 | Report with confidence | | Calibration | ECE <0.1 | Trust probabilities | ## Author AI Group - Biomedical AI Platform ## References - https://pubmed.ncbi.nlm.nih.gov/41952838/ --- Source attribution: mdbabumiamssm Original project: https://github.com/mdbabumiamssm/LLMs-Universal-Life-Science-and-Clinical-Skills-/blob/ef3c40b235ba23083e67263a921759784f0dd251/Skills/Clinical/Multimodal_Radpath_Fusion_Agent/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Dave AI. ## Upstream license notice MIT License Copyright (c) 2025 MD BABU MIA , PHD Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Dave AI resource metadata Creator / publisher: mdbabumiamssm Original project: https://github.com/mdbabumiamssm/LLMs-Universal-Life-Science-and-Clinical-Skills-/blob/ef3c40b235ba23083e67263a921759784f0dd251/Skills/Clinical/Multimodal_Radpath_Fusion_Agent/SKILL.md Attribution basis: Recovered original package and original project license at a pinned revision Fields of work: Data & Analytics, Healthcare & Life Sciences, Product Management Roles: Business Intelligence Analyst, Clinical Research, Data Scientist, Health Informatics, Product Manager Specialties: API, Backend Tools: Python Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 52f15326f898af57917497cd07382ec8af801334543f1f368b2d108ef5c70ca5 Metadata reviewed: 2026-10-27 This metadata supersedes earlier Dave AI catalog attribution. Original instruction and supporting files are preserved.