科研 Skill
科研机器学习流程
ml-for-research
Applies machine learning to research questions with scikit-learn pipelines, AutoML, SHAP explainability, and nested cross-validation. Use when building a predictive model for a study.
SKILL.md
Machine Learning for Research — Pipelines, AutoML, and Explainability
TL;DR — Build reproducible ML workflows for research: scikit-learn ColumnTransformer pipelines, nested cross-validation, FLAML AutoML with baseline comparison, Optuna hyperparameter search, SHAP TreeExplainer summaries, calibration curves, and model cards for transparent reporting.
When to Use This Skill
Use this Skill whenever you need to:
- Apply supervised ML to tabular research data (clinical, social science, omics)
- Construct a preprocessing pipeline handling mixed numeric and categorical features
- Evaluate models with nested cross-validation to obtain unbiased performance estimates
- Run AutoML (FLAML) with a time budget and compare it to a simple baseline
- Tune hyperparameters with Optuna TPE search
- Explain model predictions globally (SHAP summary) and locally (waterfall plots)
- Assess classifier calibration and apply probability calibration
- Document model properties in a structured model card for reproducible reporting
| Task | When to apply |
|---|---|
| ColumnTransformer pipeline | Any dataset with mixed feature types |
| Nested CV | Primary model evaluation for a research paper |
| FLAML AutoML | Quick strong baseline; feature selection exploration |
| SHAP TreeExplainer | Tree-based models (RF, XGBoost, LightGBM) |
| CalibratedClassifierCV | Any model used for probabilistic risk prediction |
| Model card | Methods section reporting; supplementary material |
Background & Key Concepts
scikit-learn Pipeline and ColumnTransformer
A Pipeline chains preprocessing steps and a final estimator so that
cross-validation splits are applied correctly — the scaler and imputer are fit
only on the training fold, preventing data leakage.
ColumnTransformer applies different transformers to numeric and categorical
columns in parallel, then concatenates the results:
- Numeric:
SimpleImputer(strategy='median')→StandardScaler() - Categorical:
SimpleImputer(strategy='most_frequent')→OneHotEncoder(handle_unknown='ignore')
Train / Validation / Test Split
| Split | Fraction | Purpose |
|---|---|---|
| Train | 60 % | Model fitting and cross-validation |
| Validation | 20 % | Hyperparameter selection and early stopping |
| Test | 20 % | Final unbiased performance estimate (touch once) |
Use stratify=y for classification to preserve class balance across splits.
Nested Cross-Validation
Nested CV prevents optimistic bias when tuning and evaluating on the same data:
- Outer loop:
RepeatedStratifiedKFold(n_splits=5, n_repeats=3)— provides 15 performance estimates that are averaged for the paper. - Inner loop:
GridSearchCVorRandomizedSearchCV— tunes hyperparameters on each outer training fold independently.
FLAML AutoML
FLAML (Fast and Lightweight AutoML) explores model families and hyperparameters
within a user-defined time_budget (seconds). Key parameters:
| Parameter | Recommendation |
|---|---|
time_budget | 60–300 s for initial exploration; 3600 s for final run |
task | 'classification' or 'regression' |
metric | 'roc_auc', 'accuracy', 'r2', 'rmse' |
estimator_list | ['lgbm', 'rf', 'xgboost', 'linear'] or 'auto' |
SHAP TreeExplainer
SHAP (SHapley Additive exPlanations) provides feature attributions that satisfy
desirable properties (efficiency, symmetry, dummy). TreeExplainer computes
exact SHAP values for tree-based models in polynomial time.
Key plots:
- summary_plot: global feature importance ranked by mean |SHAP|
- waterfall_plot: single-instance explanation showing additive contributions
- dependence_plot: interaction between one feature's value and its SHAP value
Model Calibration
A well-calibrated model produces probabilities that match empirical frequencies:
predicted probability 0.7 should correspond to ~70 % event rate. Use
CalibratedClassifierCV(method='isotonic') for post-hoc calibration and
CalibrationDisplay for the reliability diagram.
Environment Setup
# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install required packages
pip install "scikit-learn>=1.2" "flaml>=2.0" "shap>=0.42" \
"optuna>=3.0" "pandas>=1.5" "matplotlib>=3.6"
# Verify installation
python -c "import sklearn, flaml, shap, optuna, pandas, matplotlib; print('Setup OK')"
Optional: suppress verbose Optuna and FLAML logs during development:
export OPTUNA_VERBOSITY=WARNING # reduce Optuna console output
export FLAML_VERBOSITY=0 # 0 = silent, 1 = info, 2 = debug
Core Workflow
Step 1 — Nested CV Pipeline with SHAP Summary Plot
import os
import warnings
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import shap
from sklearn.datasets import make_classification
from sklearn.model_selection import (
RepeatedStratifiedKFold,
GridSearchCV,
train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
warnings.filterwarnings("ignore")
def build_preprocessing_pipeline(
numeric_cols: list[str],
categorical_cols: list[str],
) -> ColumnTransformer:
"""
Build a ColumnTransformer for mixed numeric and categorical features.
Args:
numeric_cols: List of numeric feature column names.
categorical_cols: List of categorical feature column names.
Returns:
Fitted ColumnTransformer (fit inside a Pipeline to avoid leakage).
"""
numeric_transformer = Pipeline([
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler()),
])
categorical_transformer = Pipeline([
("imputer", SimpleImputer(strategy="most_frequent")),
("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])
return ColumnTransformer(transformers=[
("num", numeric_transformer, numeric_cols),
("cat", categorical_transformer, categorical_cols),
], remainder="drop")
def run_nested_cv(
X: pd.DataFrame,
y: pd.Series,
numeric_cols: list[str],
categorical_cols: list[str],
outer_n_splits: int = 5,
outer_n_repeats: int = 3,
inner_cv_folds: int = 5,
param_grid: dict | None = None,
random_state: int = 42,
) -> dict:
"""
Run nested cross-validation for a RandomForestClassifier.
Args:
X: Feature DataFrame.
y: Binary target Series.
numeric_cols: Numeric feature column names.
categorical_cols: Categorical feature column names.
outer_n_splits: Number of outer CV folds.
outer_n_repeats: Number of outer CV repeats.
inner_cv_folds: Number of inner CV folds for hyperparameter tuning.
param_grid: Hyperparameter grid for GridSearchCV. Uses default if None.
random_state: Random seed.
Returns:
Dictionary with outer AUC scores list, mean AUC, and std AUC.
"""
if param_grid is None:
param_grid = {
"clf__n_estimators": [100, 200],
"clf__max_depth": [None, 5, 10],
"clf__min_samples_leaf": [1, 5],
}
preproc = build_preprocessing_pipeline(numeric_cols, categorical_cols)
clf = RandomForestClassifier(random_state=random_state, n_jobs=-1)
pipeline = Pipeline([("preproc", preproc), ("clf", clf)])
outer_cv = RepeatedStratifiedKFold(
n_splits=outer_n_splits, n_repeats=outer_n_repeats, random_state=random_state
)
inner_cv = RepeatedStratifiedKFold(n_splits=inner_cv_folds, n_repeats=1,
random_state=random_state)
outer_scores = []
for fold_idx, (train_idx, test_idx) in enumerate(outer_cv.split(X, y)):
X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
gs = GridSearchCV(
pipeline, param_grid, cv=inner_cv,
scoring="roc_auc", n_jobs=-1, refit=True,
)
gs.fit(X_train, y_train)
y_prob = gs.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_prob)
outer_scores.append(auc)
if fold_idx % 5 == 0:
print(f" Outer fold {fold_idx+1}/{outer_n_splits*outer_n_repeats}: AUC = {auc:.4f}")
mean_auc = float(np.mean(outer_scores))
std_auc = float(np.std(outer_scores, ddof=1))
print(f"\nNested CV result: AUC = {mean_auc:.4f} ± {std_auc:.4f}")
return {"outer_scores": outer_scores, "mean_auc": mean_auc, "std_auc": std_auc}
def compute_shap_values_and_plot(
X_train: pd.DataFrame,
X_test: pd.DataFrame,
y_train: pd.Series,
numeric_cols: list[str],
categorical_cols: list[str],
output_path: str = "shap_summary.png",
max_display: int = 15,
) -> None:
"""
Fit a RandomForest on X_train, compute SHAP values on X_test, and save summary plot.
Args:
X_train: Training features (DataFrame).
X_test: Test features (DataFrame).
y_train: Training labels.
numeric_cols: Numeric feature column names.
categorical_cols: Categorical feature column names.
output_path: Path to save the SHAP summary plot.
max_display: Maximum features to display in the summary plot.
"""
preproc = build_preprocessing_pipeline(numeric_cols, categorical_cols)
clf = RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)
pipeline = Pipeline([("preproc", preproc), ("clf", clf)])
pipeline.fit(X_train, y_train)
# Transform test set for SHAP
X_test_transformed = pipeline.named_steps["preproc"].transform(X_test)
feature_names = (
numeric_cols
+ list(pipeline.named_steps["preproc"]
.named_transformers_["cat"]
.named_steps["encoder"]
.get_feature_names_out(categorical_cols))
)
explainer = shap.TreeExplainer(pipeline.named_steps["clf"])
shap_values = explainer.shap_values(X_test_transformed)
# For binary classification, take class-1 SHAP values
sv = shap_values[1] if isinstance(shap_values, list) else shap_values
plt.figure(figsize=(10, 6))
shap.summary_plot(
sv,
X_test_transformed,
feature_names=feature_names,
max_display=max_display,
show=False,
plot_type="dot",
)
plt.tight_layout()
plt.savefig(output_path, dpi=150, bbox_inches="tight")
plt.close()
print(f"SHAP summary plot saved to {output_path}")
if __name__ == "__main__":
# Generate synthetic research-like classification data
X_raw, y_raw = make_classification(
n_samples=500, n_features=20, n_informative=10,
n_redundant=4, random_state=0,
)
feature_names_num = [f"lab_{i}" for i in range(15)]
feature_names_cat = [f"group_{i}" for i in range(5)]
df = pd.DataFrame(X_raw, columns=feature_names_num + feature_names_cat)
# Discretize last 5 features as fake categorical
for col in feature_names_cat:
df[col] = pd.cut(df[col], bins=3, labels=["low", "med", "high"])
target = pd.Series(y_raw, name="outcome")
X_train_full, X_test, y_train_full, y_test = train_test_split(
df, target, test_size=0.20, stratify=target, random_state=42
)
print("Running nested cross-validation...")
results = run_nested_cv(
X_train_full, y_train_full,
numeric_cols=feature_names_num,
categorical_cols=feature_names_cat,
)
print(f"\nFinal nested CV AUC: {results['mean_auc']:.3f} ± {results['std_auc']:.3f}")
print("\nComputing SHAP values...")
compute_shap_values_and_plot(
X_train_full, X_test, y_train_full,
numeric_cols=feature_names_num,
categorical_cols=feature_names_cat,
output_path="shap_summary.png",
)
Step 2 — FLAML AutoML and Comparison with Baseline
import time
import pandas as pd
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from flaml import AutoML
def run_flaml_automl(
X_train: np.ndarray | pd.DataFrame,
y_train: np.ndarray | pd.Series,
X_test: np.ndarray | pd.DataFrame,
y_test: np.ndarray | pd.Series,
time_budget: int = 60,
metric: str = "roc_auc",
task: str = "classification",
estimator_list: list[str] | None = None,
seed: int = 42,
) -> dict:
"""
Run FLAML AutoML and compare with a logistic regression baseline.
Args:
X_train: Training features.
y_train: Training labels.
X_test: Test features.
y_test: Test labels.
time_budget: Maximum seconds for FLAML search.
metric: Optimization metric (e.g., 'roc_auc', 'accuracy').
task: ML task type: 'classification' or 'regression'.
estimator_list: Model families to consider; None = auto.
seed: Random seed.
Returns:
Dictionary with AutoML and baseline AUC, best model name, and elapsed time.
"""
if estimator_list is None:
estimator_list = ["lgbm", "rf", "xgboost", "lrl2"]
# Baseline: majority class
dummy = DummyClassifier(strategy="stratified", random_state=seed)
dummy.fit(X_train, y_train)
dummy_auc = roc_auc_score(y_test, dummy.predict_proba(X_test)[:, 1])
# Logistic regression baseline
lr = LogisticRegression(max_iter=1000, random_state=seed)
lr.fit(X_train, y_train)
lr_auc = roc_auc_score(y_test, lr.predict_proba(X_test)[:, 1])
# FLAML AutoML
automl = AutoML()
settings = {
"time_budget": time_budget,
"metric": metric,
"task": task,
"estimator_list": estimator_list,
"seed": seed,
"verbose": 0,
}
t0 = time.time()
automl.fit(X_train, y_train, **settings)
elapsed = round(time.time() - t0, 1)
automl_prob = automl.predict_proba(X_test)[:, 1]
automl_auc = roc_auc_score(y_test, automl_prob)
print(f"Baseline (Dummy) AUC: {dummy_auc:.4f}")
print(f"Baseline (LogReg) AUC: {lr_auc:.4f}")
print(f"FLAML AutoML AUC: {automl_auc:.4f} "
f"[best model: {automl.best_estimator}, elapsed: {elapsed}s]")
return {
"dummy_auc": round(dummy_auc, 4),
"logreg_auc": round(lr_auc, 4),
"automl_auc": round(automl_auc, 4),
"best_estimator": automl.best_estimator,
"best_config": automl.best_config,
"elapsed_sec": elapsed,
"model": automl,
}
if __name__ == "__main__":
X, y = make_classification(n_samples=800, n_features=25, n_informative=12,
random_state=1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
stratify=y, random_state=1)
result = run_flaml_automl(X_tr, y_tr, X_te, y_te, time_budget=60)
print(f"\nAUC lift over LogReg: "
f"{(result['automl_auc'] - result['logreg_auc'])*100:+.2f} pp")
Step 3 — Model Card Generation
import json
import datetime
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.calibration import CalibratedClassifierCV, CalibrationDisplay
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
from sklearn.metrics import (
roc_auc_score, average_precision_score, brier_score_loss,
classification_report,
)
MODEL_CARD_TEMPLATE = {
"model_name": "",
"version": "1.0.0",
"date_trained": "",
"intended_use": {
"primary_use": "",
"out_of_scope": [],
},
"training_data": {
"source": "",
"n_samples": 0,
"n_features": 0,
"target_prevalence": 0.0,
"preprocessing": [],
},
"evaluation_data": {
"source": "",
"n_samples": 0,
},
"performance": {
"roc_auc": 0.0,
"average_precision": 0.0,
"brier_score": 0.0,
"calibrated": False,
},
"limitations": [],
"ethical_considerations": [],
"caveats_and_recommendations": [],
}
def generate_model_card(
model_name: str,
y_test: np.ndarray,
y_prob: np.ndarray,
training_info: dict,
limitations: list[str],
ethical_considerations: list[str],
output_path: str = "model_card.json",
) -> dict:
"""
Populate and save a model card dictionary with performance metrics.
Args:
model_name: Short name for the model.
y_test: True binary labels on the held-out test set.
y_prob: Predicted probabilities for the positive class.
training_info: Dict with keys: source, n_train, n_features,
prevalence, preprocessing.
limitations: List of known model limitations.
ethical_considerations: List of ethical considerations.
output_path: File path to save the JSON model card.
Returns:
Populated model card dictionary.
"""
import copy
card = copy.deepcopy(MODEL_CARD_TEMPLATE)
card["model_name"] = model_name
card["version"] = "1.0.0"
card["date_trained"] = datetime.date.today().isoformat()
card["training_data"]["source"] = training_info.get("source", "")
card["training_data"]["n_samples"] = training_info.get("n_train", 0)
card["training_data"]["n_features"] = training_info.get("n_features", 0)
card["training_data"]["target_prevalence"] = round(float(np.mean(y_test)), 4)
card["training_data"]["preprocessing"] = training_info.get("preprocessing", [])
card["evaluation_data"]["n_samples"] = len(y_test)
card["performance"]["roc_auc"] = round(roc_auc_score(y_test, y_prob), 4)
card["performance"]["average_precision"] = round(
average_precision_score(y_test, y_prob), 4
)
card["performance"]["brier_score"] = round(brier_score_loss(y_test, y_prob), 4)
card["limitations"] = limitations
card["ethical_considerations"] = ethical_considerations
with open(output_path, "w", encoding="utf-8") as fh:
json.dump(card, fh, indent=2)
print(f"Model card saved to {output_path}")
return card
def plot_calibration_curve(
y_test: np.ndarray,
y_prob_uncal: np.ndarray,
y_prob_cal: np.ndarray,
output_path: str = "calibration_curve.png",
) -> None:
"""Plot reliability diagram comparing uncalibrated and calibrated probabilities."""
fig, ax = plt.subplots(figsize=(7, 5))
CalibrationDisplay.from_predictions(
y_test, y_prob_uncal, n_bins=10, ax=ax, name="Uncalibrated"
)
CalibrationDisplay.from_predictions(
y_test, y_prob_cal, n_bins=10, ax=ax, name="Calibrated (isotonic)"
)
ax.set_title("Calibration Curve (Reliability Diagram)")
plt.tight_layout()
plt.savefig(output_path, dpi=150)
print(f"Calibration curve saved to {output_path}")
if __name__ == "__main__":
X, y = make_classification(n_samples=600, n_features=20,
n_informative=10, random_state=5)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
stratify=y, random_state=5)
rf = RandomForestClassifier(n_estimators=100, random_state=5)
rf.fit(X_tr, y_tr)
y_prob_uncal = rf.predict_proba(X_te)[:, 1]
cal_rf = CalibratedClassifierCV(rf, method="isotonic", cv=5)
cal_rf.fit(X_tr, y_tr)
y_prob_cal = cal_rf.predict_proba(X_te)[:, 1]
plot_calibration_curve(y_te, y_prob_uncal, y_prob_cal)
card = generate_model_card(
model_name="RandomForest-AD-Classifier",
y_test=y_te,
y_prob=y_prob_cal,
training_info={
"source": "Synthetic research dataset",
"n_train": len(X_tr),
"n_features": X_tr.shape[1],
"prevalence": float(y_tr.mean()),
"preprocessing": ["StandardScaler", "SimpleImputer(median)"],
},
limitations=[
"Trained on synthetic data; external validation required.",
"Does not account for temporal drift in feature distributions.",
],
ethical_considerations=[
"Model should not be used for clinical decision-making without "
"prospective validation.",
"Performance has not been assessed in subgroups defined by race, "
"sex, or socioeconomic status.",
],
)
print("\nModel Card:")
import json
print(json.dumps(card["performance"], indent=2))
Advanced Usage
Optuna Hyperparameter Search
import optuna
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification
optuna.logging.set_verbosity(optuna.logging.WARNING)
def optuna_objective(
trial: optuna.Trial,
X: np.ndarray,
y: np.ndarray,
cv_folds: int = 5,
) -> float:
"""
Optuna objective for RandomForest hyperparameter tuning.
Suggests n_estimators and max_depth via the trial object, fits a
RandomForest with cross-validation, and returns mean AUC.
Args:
trial: Optuna trial object.
X: Feature matrix.
y: Binary labels.
cv_folds: Number of stratified CV folds.
Returns:
Mean ROC-AUC across CV folds (to be maximized).
"""
n_estimators = trial.suggest_int("n_estimators", 50, 500, step=50)
max_depth = trial.suggest_int("max_depth", 3, 15)
min_samples_leaf = trial.suggest_int("min_samples_leaf", 1, 20)
max_features = trial.suggest_float("max_features", 0.3, 1.0)
clf = RandomForestClassifier(
n_estimators=n_estimators,
max_depth=max_depth,
min_samples_leaf=min_samples_leaf,
max_features=max_features,
random_state=42,
n_jobs=-1,
)
scores = cross_val_score(clf, X, y, cv=cv_folds, scoring="roc_auc", n_jobs=-1)
return float(scores.mean())
X, y = make_classification(n_samples=500, n_features=20, random_state=0)
study = optuna.create_study(direction="maximize",
sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(lambda trial: optuna_objective(trial, X, y), n_trials=50, show_progress_bar=False)
print(f"Best AUC: {study.best_value:.4f}")
print(f"Best params: {study.best_params}")
Permutation Feature Importance
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.inspection import permutation_importance
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=400, n_features=15, n_informative=8, random_state=3)
feature_names = [f"feature_{i:02d}" for i in range(X.shape[1])]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=3)
clf = RandomForestClassifier(n_estimators=100, random_state=3)
clf.fit(X_tr, y_tr)
perm = permutation_importance(clf, X_te, y_te, n_repeats=20,
scoring="roc_auc", random_state=3, n_jobs=-1)
perm_df = pd.DataFrame({
"feature": feature_names,
"importance_mean": perm.importances_mean,
"importance_std": perm.importances_std,
}).sort_values("importance_mean", ascending=False)
print(perm_df.head(10).to_string(index=False))
Troubleshooting
| Problem | Likely cause | Fix |
|---|---|---|
ValueError: could not convert string to float | Categorical columns passed to scaler | Ensure ColumnTransformer routes categoricals to OneHotEncoder |
SHAP TypeError: unhashable type | Non-numeric transformed features | Use get_feature_names_out() after transformation |
| FLAML timeout reached immediately | time_budget too small | Set time_budget ≥ 30 s; check system clock |
| Nested CV very slow | Too many grid combinations | Use RandomizedSearchCV instead of GridSearchCV |
| Calibration curve not flat | Overfit uncalibrated probabilities | Apply CalibratedClassifierCV(method='isotonic', cv=5) |
Optuna TrialPruned errors | Intermediate value reporting missing | Remove pruning callbacks if not using early stopping |
| Memory error in SHAP | Dataset too large for TreeExplainer | Subsample X_test to 500–1000 rows for SHAP computation |
External Resources
- scikit-learn Pipeline docs: https://scikit-learn.org/stable/modules/compose.html
- FLAML documentation: https://microsoft.github.io/FLAML/
- SHAP documentation: https://shap.readthedocs.io/
- Optuna documentation: https://optuna.readthedocs.io/
- Nested CV guide: https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html
- Model Cards for Model Reporting (Mitchell et al. 2019): https://arxiv.org/abs/1810.03993
- Calibration of classifiers: https://scikit-learn.org/stable/modules/calibration.html
Examples
Example 1 — Full Nested CV on a Clinical Dataset
import pandas as pd
from sklearn.model_selection import train_test_split
# Load your dataset (replace with real data path)
# df = pd.read_csv("clinical_data.csv")
# For demonstration, use synthetic data
from sklearn.datasets import make_classification
import numpy as np
X_raw, y_raw = make_classification(n_samples=600, n_features=18,
n_informative=9, random_state=10)
num_cols = [f"biomarker_{i}" for i in range(13)]
cat_cols = [f"category_{i}" for i in range(5)]
df = pd.DataFrame(X_raw, columns=num_cols + cat_cols)
for col in cat_cols:
df[col] = pd.cut(df[col], bins=3, labels=["A", "B", "C"])
y = pd.Series(y_raw, name="outcome")
X_tr, X_te, y_tr, y_te = train_test_split(df, y, test_size=0.2,
stratify=y, random_state=42)
nested_results = run_nested_cv(
X_tr, y_tr, numeric_cols=num_cols, categorical_cols=cat_cols,
outer_n_splits=5, outer_n_repeats=3,
)
print(f"Nested CV AUC: {nested_results['mean_auc']:.3f} ± {nested_results['std_auc']:.3f}")
Example 2 — FLAML AutoML vs Logistic Regression Baseline
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=1000, n_features=30,
n_informative=15, random_state=99)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
stratify=y, random_state=99)
result = run_flaml_automl(X_tr, y_tr, X_te, y_te, time_budget=60)
print(f"\nBest model: {result['best_estimator']}")
print(f"AUC improvement over LogReg: "
f"{(result['automl_auc'] - result['logreg_auc'])*100:+.2f} pp")
Example 3 — Model Card with Calibration
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.calibration import CalibratedClassifierCV
X, y = make_classification(n_samples=700, n_features=20, random_state=7)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
stratify=y, random_state=7)
base = RandomForestClassifier(n_estimators=100, random_state=7)
cal = CalibratedClassifierCV(base, method="isotonic", cv=5)
cal.fit(X_tr, y_tr)
y_prob = cal.predict_proba(X_te)[:, 1]
card = generate_model_card(
model_name="CalibratedRF-v1",
y_test=y_te,
y_prob=y_prob,
training_info={
"source": "Demo dataset",
"n_train": len(X_tr),
"n_features": 20,
"preprocessing": ["StandardScaler"],
},
limitations=["Synthetic data only; requires real-world validation."],
ethical_considerations=["No subgroup fairness analysis performed."],
output_path="model_card_demo.json",
)
import json
print(json.dumps(card["performance"], indent=2))
Changelog
| Version | Date | Change |
|---|---|---|
| 1.0.0 | 2026-03-18 | Initial release — nested CV, FLAML AutoML, SHAP, calibration, Optuna, model cards |
示例提示词
帮我用scikit-learn搭建一个带嵌套交叉验证和SHAP解释的预测模型。
所属插件
安装该插件会一并获得同领域的 54 个 Skill / MCP。
同插件的其他科研 Skill
- 智能体资源节约守则Applies token- and context-saving habits for tool choice, PDF handling, persistent memory, and privacy checks. Use at the start of sessions with many files or PDFs, or multi-session work.
- 英文论文双语精读器把英文论文转成中英对照精读档并生成离线双语阅读器,标注要点并输出读书笔记。精读单篇英文论文、要中英对照阅读时使用。
- 研究方法比较论证Compares quantitative, qualitative, and mixed research methods against what each can and cannot answer to build a methods table. Use when justifying a method choice in a thesis.
- 复杂系统分析方法Analyzes complex systems with power-law fitting, Hurst exponent, fractal dimension, agent-based modeling, and entropy measures. Use for nonlinear or complex-system data.
- 研究环境容器化部署Builds Docker and Singularity containers for reproducible computation and HPC deployment of research environments. Use when packaging an analysis pipeline for reuse or HPC.
- DVC数据版本管理Sets up DVC for dataset and pipeline versioning, remote storage, and experiment comparison in reproducible ML research workflows. Use when tracking large datasets or ML experiment runs.
- 期刊级统计图表绘制Creates publication-quality multi-panel figures with matplotlib, seaborn, or ggplot2 using colorblind-safe palettes and journal export settings. Use when preparing figures for a paper.
- 综述模块拆解规划Breaks a refined literature-review question into 3-5 non-overlapping modules to build a reading map, outline, or search plan. Use when structuring a literature review.
- 学位论文答辩教练模拟学位口试委员提问并提供承认回应证据三段式答辩框架与模拟问答。准备论文口试或研讨会问答环节时使用。
- 实验设计与样本量规划Designs rigorous experiments covering sample-size calculation, randomization strategy, pre-registration, and A/B test duration estimation. Use when planning an experiment or A/B test.
- 管理实验设计架构为管理与商学情境实验规划组间组内设计、对抗平衡与操弄检核,并产出设计书与情境材料模板。做商学实验设计时使用。
- 科研经费申请撰写Structures research grant proposals including NSF/NIH Specific Aims, budget justification, ERC narrative sections, and biosketch formatting. Use when drafting a funding application.
- 人类可读文档撰写规范Guides writing human-facing docs like README, PLAN, survey notes, and reviewer-response drafts with a clear two-tier structure. Use for text a human collaborator will read.
- 知识图谱与SPARQL检索Builds knowledge graphs and runs SPARQL queries over RDF and Wikidata for entity linking in research and library data. Use when modeling research entities as a linked graph.
- LaTeX论文写作与排版Writes, compiles, and submits LaTeX papers covering IMRaD structure, key packages, bibliography management, and arXiv preparation. Use when authoring or debugging a LaTeX manuscript.
- 文献比较矩阵构建器把一批PDF整理成含撤稿查核与APA7书目的文献资料库,并生成横向比较矩阵表。整理综述前要比较文献时使用。
- 跨库学术文献检索Searches OpenAlex, Semantic Scholar, and arXiv for academic literature with deduplication, citation snowballing, and BibTeX export. Use when collecting papers for a review or topic.
- 文献调研语料库维护Finds, verifies, and maintains a literature corpus from arXiv, OpenAlex, Crossref, and related databases with BibTeX entries and per-paper notes. Use when sourcing references for a paper.
- 文献综述研究缺口定位Converts a literature-review synthesis into a concrete, defensible research gap and future-research direction. Use when writing a conclusion or thesis contribution section.
- 文献矛盾结论梳理Analyzes conflicting findings across papers and explains disagreement by sample, method, or context differences for a stronger synthesis. Use when review sources disagree.
- 效应量汇总元分析Pools effect sizes across studies with fixed and random-effects models, heterogeneity tests, forest and funnel plots, and subgroup analysis. Use to synthesize quantitative study results.
- 缺失数据处理Diagnoses missing-data mechanisms such as MCAR, MAR, or MNAR and applies multiple imputation or deletion with sensitivity analysis. Use when a dataset has missing values.
- 混合方法研究设计Designs mixed-methods studies integrating qualitative coding, triangulation, content analysis, and QCA with inter-rater reliability checks. Use when combining qual and quant data.
- 开放数据存储与FAIRDeposits and documents research data via Zenodo or OSF with FAIR assessment, data management plans, and DataCite metadata. Use when publishing or sharing a research dataset.
常见问题
科研机器学习流程能做什么?
Applies machine learning to research questions with scikit-learn pipelines, AutoML, SHAP explainability, and nested cross-validation. Use when building a predictive model for a study。
如何安装科研机器学习流程?
在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「科研机器学习流程」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。
科研机器学习流程需要哪些权限?
该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。
科研机器学习流程经过官方审核吗?
是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。