In May 2024, a research team led by the Massachusetts Institute of Technology (MIT) published a landmark paper in Nature Biotechnology outlining the first comprehensive benchmarking framework for foundation models in chemistry (FMCs). This study represents a crucial step forward in systematizing the evaluation of AI models applied to chemical and molecular sciences.
Foundation models—large-scale machine learning systems pretrained on massive datasets—have transformed fields like natural language processing (NLP) and computer vision. However, their impact in chemistry has been harder to quantify due to the diversity of chemical tasks and representations. To address this gap, the authors introduced an open, modular benchmarking infrastructure spanning six major task domains and 39 curated datasets.
Scope and Design of the Benchmark
The benchmark focuses on six primary domains relevant to chemical AI:
- Molecular Property Prediction – such as toxicity or solubility.
- Reaction and Retrosynthesis Prediction – used in organic chemistry and pharmaceutical pipelines.
- Protein Modeling – including structural and functional tasks like secondary structure prediction.
- Molecular Generation & Optimization – for drug and material discovery.
- Cross-modal Alignment – aligning chemical data with natural language.
- Protein–Ligand Binding – a critical task for structure-based drug design.
Datasets were sourced from widely adopted public benchmarks including MoleculeNet, TAPE, PDBBind, and OpenCatalyst, and harmonized for consistent preprocessing and labeling formats. Each dataset was mapped to a specific task type (classification, regression, or ranking), with domain-specific metrics applied.
Models and Architectures Evaluated
The team evaluated a total of 84 model checkpoints, contributed by 26 teams worldwide. These spanned a wide variety of architectures, including:
- Graph Neural Networks (GNNs) – processing molecules as node–edge graphs.
- Sequence Models (e.g., Transformers) – treating SMILES strings like text.
- Multimodal Models – combining chemical graphs, textual metadata, or protein sequences/images.
Well-known models such as GROVER, ChemBERTa, MolFormer, OpenCatalyst, and ESM were included, alongside lesser-known and experimental architectures. The framework utilized a standardized interface called ModelAdapter, which enabled model-specific wrappers while maintaining consistent input/output pipelines.
Evaluation Methodology
Each model was subjected to three evaluation settings:
- Zero-shot testing – no task-specific fine-tuning, testing model generalization.
- Fine-tuned performance – training the model on the task dataset.
- Transfer learning – pretraining on related tasks followed by targeted fine-tuning.
Metrics used included:
- MAE / RMSE / R² for regression tasks,
- ROC-AUC / PR-AUC for classification,
- Spearman / Kendall correlations for ranking problems.
Models were evaluated multiple times using random seeds to ensure reproducibility. Statistical comparisons and significance tests were performed to validate performance differences across architectures and input modalities.
Key Findings
- No single model dominated across all task domains. Performance was highly task- and domain-specific.
- Input modality matters: Models trained on molecular graphs often outperformed SMILES-based models for structure-sensitive tasks, while sequence models excelled in natural language-aligned benchmarks.
- Pretraining showed mixed results: While beneficial in protein-based tasks (e.g., ESM), some GNNs performed well even without pretraining.
- Finetuning strategies matter: Frozen encoders versus fully-trainable architectures yielded different outcomes depending on the model-task fit.
- Multimodal models remain early-stage: While promising, they have not yet demonstrated consistent improvements over uni-modal models.
Call to Action
The authors emphasize the urgent need for an open-access evaluation platform for chemical foundation models—akin to Hugging Face or OpenML for other AI disciplines. Such a platform would ensure transparency, encourage reproducibility, and promote healthy competition in the field.
Additionally, they advocate for continually updated benchmarks, particularly in underrepresented chemical tasks such as reaction condition prediction, low-resource bioassays, and rare molecule design.
Implications for the Future
This work signals a new era for AI in chemistry, where progress can be systematically tracked and compared. As foundation models increasingly power applications in drug discovery, materials design, and protein engineering, a robust benchmarking ecosystem will be essential to channel innovation responsibly.
It also opens the door for more interdisciplinary collaboration—connecting model developers, chemists, biologists, and pharmaceutical researchers through a common evaluation language.