科研 Skill
小样本材料化学机器学习
small-data-ml
Covers machine learning on small scientific datasets (roughly 20-2000 samples) in materials science and chemistry: leakage-safe splitting. Use when training ML on a small dataset.
SKILL.md
Small-Data Machine Learning for Materials and Chemistry
Description
This skill covers machine learning on small scientific datasets (roughly 20–2000 samples) in materials science and chemistry: leakage-safe splitting strategies for correlated structures and compositions, baseline model selection, descriptor screening, uncertainty quantification, learning curves, model selection under limited data, transfer learning from pretrained representations, and rigorous reporting of performance including confidence intervals and negative results. Invoke this skill when building a property predictor, surrogate model, or screening model from a dataset small enough that every methodological choice — splitting strategy, baseline, hyperparameter tuning procedure — materially affects the reported metrics.
Domain Context
Most published materials ML models are trained on fewer than a few thousand samples. This is not a weakness — DFT, synthesis, and characterization are expensive — but it creates methodological traps that are easy to miss and hard to detect from the outside. High R² on a random 80/20 split of a 200-sample dataset is not evidence of a useful model; it may reflect correlation leakage, a narrow target range, or an over-tuned hyperparameter.
Why scientific small-data is different from standard ML small-data:
In general ML, small-data problems usually assume i.i.d. samples. Materials datasets violate this in structured ways:
- Compositional correlation: Two alloys that differ by 1 at% are nearly identical in descriptor space and probably in target space. Random splitting separates near-duplicates across train and test, making the model look more accurate than it generalizes.
- Structural prototype correlation: All rocksalt oxides share the same topology; a model trained on MgO, NiO, CoO will trivially interpolate to FeO even under random splitting.
- Database-sourced duplicates: Materials Project, AFLOW, and OQMD share many structures computed by different methods; deduplication before splitting is mandatory.
- Experimental campaign structure: Samples in a synthesis campaign often share substrate, precursor, or equipment — a group that must be held out together, not split randomly.
- Active-learning iteration structure: Samples from later iterations are in regions near those from earlier iterations. Splitting across iterations pretends the model was trained on future knowledge.
Signal-to-noise ratio: DFT targets are exact given the settings (no measurement noise), but carry systematic errors (functional, ENCUT, k-mesh). Experimental targets carry both measurement noise and sample-to-sample variability. A model whose MAE is less than the experimental measurement uncertainty is not doing better than a mean predictor — it is fitting measurement noise.
Extrapolation vs interpolation: Small datasets typically do not cover the full composition, structure, or synthesis-condition space. A model that interpolates within the training hull may look excellent but extrapolate catastrophically. The deployment regime (same chemistry? new chemistry family? new element? different processing?) must be stated and tested, not assumed.
Baseline models are not optional: Ridge regression, random forests, and k-nearest-neighbor with good descriptors often perform comparably to neural networks on small materials datasets. If a complex model does not beat simple baselines by a statistically meaningful margin, the complexity adds no value. [EXPERT REVIEW NEEDED: threshold beyond which neural networks consistently outperform GPs/RF on materials small-data is dataset-specific; 500–2000 samples is a common empirical transition zone but it varies with descriptor quality]
When to Use This Skill
- Building a property surrogate or screening model from a dataset of 20–2000 structures, compositions, or experimental conditions.
- Auditing a published or inherited materials ML model for methodological soundness before extending or citing it.
- Choosing a splitting strategy for a dataset with known correlation structure (same prototype, same campaign, same active-learning iteration).
- Diagnosing whether a high R² or low MAE reflects genuine generalization or a data artifact.
- Setting up nested cross-validation for rigorous model selection when a hold-out test set is not available.
- Determining whether the current dataset is large enough to support a reliable model, or whether more data are needed.
When Not to Use This Skill
- Large datasets (> ~5000 samples): With enough data, random splits approximate i.i.d., learning curves saturate, and neural networks become competitive. Use standard ML validation practices.
- When the target is a trajectory or time series: Correlation structure is temporal; use specialized time-series CV (forward chaining), not group-based spatial splitting.
- When data quality has not been audited: Validation techniques produce misleading metrics on datasets with unit errors, unconverged DFT, mixed functionals, or undisclosed duplicates. Audit provenance before modeling.
- When the dataset has fewer than ~20 samples: Below ~20 samples, any train/test split is statistically unreliable. Use physical reasoning, Bayesian priors, or transfer learning exclusively; do not report ML performance metrics.
- When the scientific claim requires extrapolation to a new chemical space and no held-out examples from that space exist: Validation on interpolating test sets cannot support extrapolation claims. Collect or use existing data from the target space.
Core Concepts
Full steps for each part are in references/core-concepts/; open the one you need:
- Dataset Size and Regime Classification —
references/core-concepts/01-dataset-size-and-regime-classification.md: These boundaries are rough. Dataset effective size also depends on target variance, descriptor quality, and correlation structure. [EXPERT REVIEW NEEDED] - Correlation Leakage in Materials Splits —
references/core-concepts/02-correlation-leakage-in-materials-splits.md: The most consequential methodological error in materials ML is splitting a correlated dataset as if it were i.i.d. Types of leakage: - Splitting Strategies —
references/core-concepts/03-splitting-strategies.md: Random split: Only valid when the dataset has been deduplicated, prototype-checked, and confirmed to be compositionally diverse. Report the random seed and the fraction; run multiple seeds and report variance. - Baseline Models for Small Scientific Data —
references/core-concepts/04-baseline-models-for-small-scientific-data.md: Always compute these before investing in a complex model: - Uncertainty Quantification —
references/core-concepts/05-uncertainty-quantification.md: GP posterior variance: Exact under the model; calibrate against held-out data (reliability diagram). Do not trust GP uncertainty in regions far from training data without empirical calibration. - Learning Curves and Data Sufficiency —
references/core-concepts/06-learning-curves-and-data-sufficiency.md: A learning curve plots validation error vs. training set size (using subsets of training data, not the test set). Diagnostic patterns: - Descriptor Selection and Scaling —
references/core-concepts/07-descriptor-selection-and-scaling.md: Descriptor selection in small-data settings requires care to avoid leakage:
Key Workflows
Full steps for each part are in references/key-workflows/; open the one you need:
- Workflow 1: Leakage-Safe Splitting with Group and Prototype Awareness —
references/key-workflows/01-workflow-1-leakage-safe-splitting-with-group-and.md - Workflow 2: Baseline Model Comparison —
references/key-workflows/02-workflow-2-baseline-model-comparison.md - Workflow 3: Learning Curves and Data Sufficiency —
references/key-workflows/03-workflow-3-learning-curves-and-data-sufficiency.md - Workflow 4: Uncertainty Calibration and Conformal Prediction —
references/key-workflows/04-workflow-4-uncertainty-calibration-and-conformal.md - Workflow 5: Descriptor Selection with Leakage Protection —
references/key-workflows/05-workflow-5-descriptor-selection-with-leakage-pro.md - Workflow 6: Transfer Learning from Pretrained Representations —
references/key-workflows/06-workflow-6-transfer-learning-from-pretrained-rep.md - Workflow 7: Rigorous Reporting — Confidence Intervals and Negative Results —
references/key-workflows/07-workflow-7-rigorous-reporting-confidence-interva.md
Common Failure Modes
| Failure Mode | Symptoms | Likely Cause | Diagnostic | Fix |
|---|---|---|---|---|
| Random split over correlated structures | R² near 1.0 on test; model fails on new chemistry | Near-duplicate structures in train and test; prototype or composition correlation | Switch to LOGO-CV; check R² drop; plot pairwise descriptor distances between train/test | Use LOGO-CV or LOCO-CV; deduplicate before splitting |
| Hidden duplicates | Test MAE near zero; model perfectly "predicts" test structures | Same structure appears under different IDs or slightly different cells | Run StructureMatcher on full dataset before splitting | Deduplicate with pymatgen.analysis.structure_matcher |
| Descriptor leakage | Validation R² inflated; model uses information unavailable at prediction time | Scaler, PCA, or feature selector fit on all data before splitting | Check: is any transformation fitted before the split? | Wrap all transformations in a Pipeline; fit only inside training fold |
| Target leakage | Near-perfect validation; feature correlated with target for non-causal reasons | A feature encodes the target (e.g., using experimental band gap to predict DFT band gap) | Correlate each feature with target; check feature definitions | Remove leaking features; audit feature definitions against the prediction task |
| Over-tuned hyperparameters | Validation MAE low; independent test MAE much higher | Hyperparameters tuned on the validation fold; same fold used for selection and reporting | Check: was the test fold touched during tuning? | Nested CV; treat test fold as final and untouched until reporting |
| High R² from narrow target range | R² = 0.97, but model is barely better than mean predictor | Target has low variance; R² amplifies small improvements in low-variance settings | Check skill score vs mean predictor; inspect target distribution | Report MAE, skill score, and target std alongside R²; do not rely on R² alone |
| Extrapolation without acknowledgment | Model applied to new chemistry family with confident predictions; physical nonsense results | No held-out samples from the target chemistry; validation only covers interpolation | Compute descriptor-domain coverage; check distances of new points from training hull | Flag all out-of-domain predictions; use GP or conformal intervals that widen in extrapolation |
| Confusing statistical accuracy with physical validity | Model passes CV metrics but predicts negative band gaps, wrong phase stabilities, or violated sum rules | CV metrics only measure statistical fit, not physical correctness | Add physical sanity checks: bounds, signs, limiting cases, known constraints | Implement domain-specific physical validation checks alongside statistical metrics |
| Cherry-picked metric or split | Paper reports best metric across many seeds, splits, or metrics tried | Implicit multiple comparisons; only the best result reported | Count how many splits, seeds, and metrics were tried before reporting | Pre-register the evaluation protocol; report mean ± std across seeds; correct for multiple comparisons |
| Learning curve not computed | Data sufficiency not established; possible to collect more data cheaply | Effort focused on model tuning rather than data quality assessment | Generate learning curve; check if error has plateaued | Run learning curve on training subsets; if not saturated, collect more data before tuning model |
Best Practices
Data before modeling:
- Deduplicate structures using
StructureMatcherbefore any split. A dataset with hidden duplicates will give inflated metrics regardless of the model. - Audit target provenance: mixed DFT functionals, different convergence settings, or mixed experimental conditions produce a multi-modal target distribution that single-model regression cannot handle.
- Check target distribution: is it unimodal? Are there outliers? What is the physical range? Log-transform if the target spans orders of magnitude (e.g., ionic conductivity, catalytic rate).
Splitting:
- Default to group-aware splitting (LOGO-CV or LOCO-CV) for any materials dataset. Random splits are valid only when the dataset is verified to be compositionally and structurally diverse with no cross-split correlations.
- Assign groups before looking at the target values. Do not choose groups post-hoc to produce a desired result.
- The test fold must not be used for any model decision. Nested CV enforces this automatically.
Baselines:
- Always report mean predictor, ridge regression, and random forest alongside any complex model. The complex model must beat all three by a statistically significant margin (Wilcoxon, n_folds ≥ 6) to be considered an advance.
- Run multiple random seeds for any random component (forest, bootstrap, random initialization) and report mean ± std.
Metrics:
- Report MAE (or RMSE) alongside R². R² alone is uninterpretable without knowing the target variance.
- Report skill score: 1 − MAE / MAE_mean_predictor. Values < 0.2 indicate the model is barely useful.
- Report 95% bootstrap CIs for all metrics. A difference of MAE ± 0.05 vs 0.04 is not meaningful without intervals.
- For uncertainty-aware models (GP, ensemble), report calibration coverage (e.g., "90% of test points fall within the 90% predictive interval").
Reporting negative results:
- If a complex model does not beat a simple baseline, report it. Negative results prevent repeated effort in the community.
- If the learning curve shows that the model is bias-limited (more data will not help), report the descriptor or model limitation.
- If the model fails physical sanity checks despite acceptable MAE, report both the metric and the failure. Statistical accuracy and physical validity are independent.
Integration with Other Skills
Featurization and data sources:
- Materials Featurization — descriptor pipelines (Magpie, SOAP, MBTR, graph fingerprints) that feed the feature matrix for all models in this skill. Feature selection, correlation filtering, and scaling must be applied after splitting, not before.
- Materials Databases Access — small datasets are often derived from Materials Project, AFLOW, or OQMD queries. Deduplication, compatibility checking, and provenance metadata from that skill are prerequisites for safe splitting.
Modeling and uncertainty:
- Gaussian Process Materials — GP regression is the preferred baseline model for small materials datasets where calibrated uncertainty is needed. Kernel selection, noise modeling, and calibration workflows from the GP skill are used directly in Workflows 4 and 6.
- Bayesian Optimization — BO uses the small-data surrogate as its model; the validation protocol here determines whether the surrogate's uncertainty is trustworthy enough to use as an acquisition signal. Always validate the surrogate before deploying it in a BO loop.
- Surrogate Validation — all validation concepts in this skill (LOGO-CV, calibration, learning curves, baseline comparison) directly feed into the surrogate validation skill. The surrogate validation skill applies these in the specific context of screening and optimization loops.
- Uncertainty MLP — ensemble-based and evidential uncertainty quantification for neural MLP surrogates follow the same calibration and reliability-diagram protocol introduced here for GPs and simple ensembles.
Data generation and active learning:
- High-Throughput DFT — small materials datasets often grow through iterative HT-DFT campaigns. The learning curve workflow here determines when to stop collecting data; the convergence tier policy there determines data quality.
- Active Learning MLP — active learning exploits the surrogate model built under this skill's protocol. Iteration-aware splits (never mixing training and future query iterations in validation) are essential for honest evaluation of closed-loop campaigns.
Visualization and reproducibility:
- Scientific Visualization — parity plots, residual plots, calibration curves, learning curves, and descriptor coverage maps are the standard outputs of small-data model evaluation; this skill provides the plot conventions and multi-panel figure standards.
- Reproducibility Workflow — small-data ML studies require archiving the exact dataset (with IDs and provenance), the train/val/test split indices or group assignments, the fitted scaler and selector, and the CV results. Without this, results cannot be reproduced or audited.
Key References
- Meredig, B., Agrawal, A., Kirklin, S., Saal, J. E., Doak, J. W., Thompson, A., Zhang, K., Choudhary, A. & Wolverton, C. (2014). "Combinatorial screening for new materials in unconstrained composition space with machine learning." Physical Review B 89, 094104. Early materials ML benchmark illustrating composition-based descriptors and dataset construction.
- Ramprasad, R., Batra, R., Pilania, G., Mannodi-Kanakkithodi, A. & Kim, C. (2017). "Machine learning in materials informatics: Recent applications and prospects." npj Computational Materials 3, 54. Review of descriptor families, model selection, and validation practices in materials ML.
- Schutt, K. T., Arbabzadah, F., Chmiela, S., Muller, K. R. & Tkatchenko, A. (2017). "Quantum-chemical insights from deep tensor neural networks." Nature Communications 8, 13890. Establishes neural network baselines; useful for comparing simple vs complex model performance on small molecular datasets.
- Lones, M. A. (2021). "How to avoid machine learning pitfalls: A guide for academic researchers." arXiv:2108.02497. Accessible guide to leakage, overfitting, cherry-picking, and reporting bias applicable to any scientific ML setting.
- Cawley, G. C. & Talbot, N. L. C. (2010). "On over-fitting in model selection and subsequent selection bias in performance evaluation." Journal of Machine Learning Research 11, 2079–2107. Foundational paper on nested cross-validation and model-selection bias; required reading for small-data ML.
- Angelopoulos, A. N. & Bates, S. (2023). "Conformal prediction: A gentle introduction." Foundations and Trends in Machine Learning 16, 494–591. Tutorial on conformal prediction for distribution-free coverage-guaranteed intervals.
- Vovk, V., Gammerman, A. & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer. Foundational text on conformal prediction and transductive methods for small datasets.
- Barber, R. F., Candes, E. J., Ramdas, A. & Tibshirani, R. J. (2023). "Conformal prediction beyond exchangeability." Annals of Statistics 51, 816–845. Extension of conformal prediction to correlated (non-exchangeable) scientific datasets.
[EXPERT REVIEW NEEDED: The dataset size thresholds in the regime table (20/100/500/2000) are based on practitioner experience and published benchmarks in materials ML; they are not universal and depend heavily on descriptor quality, target complexity, and the degree of correlation structure. Verify whether the Wilcoxon test requires ≥6 folds specifically or whether alternatives (paired t-test, sign test) are preferable for very few groups. Confirm whether CHGNet, MACE, and M3GNet embeddings as frozen descriptors consistently outperform Magpie on small materials datasets across multiple benchmarks, as the evidence is still accumulating.]
示例提示词
我只有不到200个样本的数据集,帮我选择防泄漏的划分方式建模。
所属插件
安装该插件会一并获得同领域的 42 个 Skill / MCP。
同插件的其他科研 Skill
- 贝叶斯优化材料发现Covers Bayesian optimization for materials and chemistry design: surrogate models, acquisition functions, batch/constrained BO. Use when planning sample-efficient experiments or simulations.
- CALPHAD热力学计算工作流Covers CALPHAD computational thermodynamics: database selection, equilibrium/metastable calculations, binary/ternary phase diagrams. Use when predicting phase stability or solidification paths.
- 材料第一性原理与分子动力学模拟Routes first-principles and classical materials simulation (DFT, MD, ML potentials) to the right specialized workflow. Use as an entry point for any simulation-based materials task.
- 蠕变力学建模与验证Selects, calibrates, and validates creep deformation and rupture models under sustained thermomechanical load, covering rate prediction. Use when assessing long-term high-temperature components.
- 晶体塑性有限元建模Covers crystal plasticity and CPFE modeling of metals: slip systems for FCC/BCC/HCP, Schmid factors. Use when simulating texture- and orientation-dependent plastic deformation.
- 衍射结构表征Covers diffraction-based structure characterization (XRD, neutron, SAED): space group determination, Rietveld refinement, phase identification. Use for any diffraction-pattern analysis technique.
- EBSD织构分析Analyzes EBSD orientation maps for grain size, misorientation, phase, and texture, with pole figures and twin-boundary identification. Use for electron backscatter diffraction workflows.
- 电化学性能测试分析Covers electrochemical characterization: cyclic voltammetry, battery cycling analysis, EIS. Use for electrochemistry-specific data analysis distinct from general materials characterization.
- 金属疲劳寿命评估Builds, calibrates, and validates fatigue assessments for metals and components: stress-life, strain-life, multiaxial critical-plane. Use when predicting fatigue life or crack growth.
- 多物理场有限元建模Covers finite element modeling for coupled materials, mechanics, and transport problems: weak-form construction, boundary conditions, mesh choice. Use for multiphysics FEM simulations.
- 连续尺度有限元力学建模Covers continuum-scale mechanical, thermal, and multiphysics simulation with Abaqus/COMSOL conventions, mesh generation. Use for macroscale FEM consuming measured or computed material properties.
- 断裂力学评估与建模Selects, constructs, and validates fracture-mechanics assessments: LEFM checks, elastic-plastic fracture mechanics. Use when assessing cracked materials or components.
- 高斯过程材料回归建模Covers Gaussian process regression for materials and chemistry datasets: descriptor selection, kernel design. Use when a probabilistic surrogate model with uncertainty is needed.
- 自驱动实验室闭环材料发现Covers combinatorial synthesis, robotic experimentation. Use for autonomous or robotics-driven experimental campaigns, a less standardized domain.
- 材料数据库可复现访问Covers reproducible API access to materials databases (Materials Project, AFLOW, OQMD, NOMAD, COD, ICSD-style sources): authentication. Use when programmatically querying these databases.
- 材料数据库跨库查询Covers querying public materials databases beyond Materials Project: AFLOW, OQMD, NOMAD, JARVIS. Use when pulling or comparing data from these live, API-integrated resources.
- 材料数据标准与互操作Covers data interoperability standards for materials science: CIF crystal files, the OPTIMADE query API. Use when structuring or exchanging materials data between tools.
- 材料结构特征化Covers converting structures, compositions, and trajectories into ML feature vectors: Magpie-style elemental statistics. Use when building inputs for materials ML models.
- 材料科学图表绘制规范Covers publication-quality figure conventions for materials science plots: phase diagrams, pole figures, stress-strain curves. Use when the deliverable is a figure, not just an analysis result.
- 材料信息学性质预测Covers ML for materials property prediction and generative design: descriptor-based models, crystal GNNs (CGCNN/MEGNet-class). Use for materials informatics distinct from interatomic potentials.
- 材料生命周期评估Covers life-cycle assessment for materials and manufacturing routes: goal/scope definition, functional units, inventory construction. Use for environmental-impact studies of materials.
- 材料科学文献检索Covers formulating literature search queries for materials science questions and synthesizing/citing results responsibly. Requires a web search tool; it cannot substitute for one.
- 工程材料选型设计Selects engineering materials via material indices and Ashby charts, ranking candidates by performance per cost or mass. Use when picking a material class for a design constraint.
- 材料统计方法与实验设计Covers statistics shared across materials characterization and simulation: design of experiments. Use for cross-cutting statistical treatment of materials data.
常见问题
小样本材料化学机器学习能做什么?
Covers machine learning on small scientific datasets (roughly 20-2000 samples) in materials science and chemistry: leakage-safe splitting. Use when training ML on a small dataset。
如何安装小样本材料化学机器学习?
在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「小样本材料化学机器学习」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。
小样本材料化学机器学习需要哪些权限?
该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。
小样本材料化学机器学习经过官方审核吗?
是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。