科研 Skill
研究环境容器化部署
containerization-research
Builds Docker and Singularity containers for reproducible computation and HPC deployment of research environments. Use when packaging an analysis pipeline for reuse or HPC.
SKILL.md
Containerization for Research
When to Use This Skill
Use this skill when you need to:
- Freeze a research computational environment for reproducibility
- Deploy analyses to HPC clusters using Singularity/Apptainer
- Automate multi-container research workflows with Docker Compose
- Share an exact computational environment with collaborators
- Build reproducible research compendiums (data + code + environment)
- Run GPU-accelerated experiments in isolated containers
- Create research environment images for cloud computing (AWS, GCP)
Trigger keywords: Docker, Dockerfile, container, reproducible research, Singularity, Apptainer, Docker Compose, HPC container, research compendium, environment isolation, image, registry, Docker Hub, container orchestration, GPU container, NVIDIA Container Toolkit, conda-pack, conda-forge, reprozip, research environment, FAIR data.
Background & Key Concepts
Container vs. Virtual Machine
Containers share the host OS kernel (lightweight, fast startup) while VMs run a complete guest OS. For research:
- Docker: standard for development and cloud; root daemon (security concern on HPC)
- Singularity/Apptainer: designed for HPC, runs as user (no root daemon), can run Docker images
Dockerfile Best Practices
- Use specific base image tags (e.g.,
python:3.11.5-slim) notlatest - Layer caching: put infrequently changing layers first (OS packages before pip installs)
- Multi-stage builds: separate build dependencies from runtime
- Non-root user:
RUN useradd -m researcher && USER researcher .dockerignore: exclude large data files and build artifacts
Image Size Optimization
- Use
python:3.11-slimorpython:3.11-alpineinstead of fullpython:3.11 - Combine
RUNcommands to reduce layers - Delete package caches:
rm -rf /var/lib/apt/lists/* - Use
--no-install-recommendswith apt
Reproducibility Guarantee
Pin all versions:
python==3.11.5
numpy==1.24.3
pandas==2.0.3
Or use lock files: pip-compile → requirements.txt, conda-lock → conda-lock.yml
Environment Setup
# Install Docker (Linux)
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Install Docker Compose
docker compose version
# Python Docker SDK
pip install docker>=6.1
# Verify
docker --version && docker compose version
Core Workflow
Step 1: Dockerfile for Research Python Environment
# File: Dockerfile
# Multi-stage research environment
# ─── Stage 1: Builder ─────────────────────────────────────────────────────────
FROM python:3.11.5-slim AS builder
WORKDIR /build
# Install build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
g++ \
gfortran \
libopenblas-dev \
&& rm -rf /var/lib/apt/lists/*
# Copy requirements first for layer caching
COPY requirements.txt .
# Install Python packages into /install
RUN pip install --prefix=/install --no-cache-dir -r requirements.txt
# ─── Stage 2: Runtime ─────────────────────────────────────────────────────────
FROM python:3.11.5-slim AS runtime
LABEL maintainer="research@university.edu"
LABEL description="Reproducible research environment for quantitative analysis"
LABEL version="1.0.0"
# Runtime system dependencies only
RUN apt-get update && apt-get install -y --no-install-recommends \
libopenblas0 \
curl \
&& rm -rf /var/lib/apt/lists/*
# Copy installed packages from builder
COPY --from=builder /install /usr/local
# Create non-root user
RUN useradd -m -u 1000 researcher
WORKDIR /workspace
RUN chown researcher:researcher /workspace
USER researcher
# Copy research code
COPY --chown=researcher:researcher src/ ./src/
COPY --chown=researcher:researcher scripts/ ./scripts/
COPY --chown=researcher:researcher config/ ./config/
# Expose Jupyter port
EXPOSE 8888
# Default command: start Jupyter Lab
CMD ["jupyter", "lab", "--ip=0.0.0.0", "--port=8888", \
"--no-browser", "--NotebookApp.token=''"]
# File: requirements.txt
numpy==1.24.4
pandas==2.0.3
scipy==1.11.4
scikit-learn==1.3.2
statsmodels==0.14.1
matplotlib==3.7.3
seaborn==0.13.0
jupyter==1.0.0
jupyterlab==4.0.9
# File: .dockerignore
.git
.github
__pycache__
*.pyc
*.pyo
*.egg-info
.pytest_cache
.coverage
htmlcov/
dist/
build/
data/*.csv
data/*.parquet
!data/sample_data.csv
*.log
.env
# Build the image
docker build -t research-env:1.0.0 .
docker build --platform linux/amd64 -t research-env:1.0.0 . # for cross-platform
# Run interactively
docker run --rm -it \
-v $(pwd)/data:/workspace/data \
-v $(pwd)/results:/workspace/results \
-p 8888:8888 \
research-env:1.0.0 bash
# Run a specific analysis script
docker run --rm \
-v $(pwd)/data:/workspace/data:ro \
-v $(pwd)/results:/workspace/results \
research-env:1.0.0 \
python src/analysis.py --config config/params.yaml
# Run Jupyter Lab
docker run --rm -d \
-v $(pwd):/workspace \
-p 8888:8888 \
--name research-jupyter \
research-env:1.0.0
Step 2: Docker Compose for Multi-Service Research Workflow
# File: docker-compose.yml
version: "3.9"
services:
# ── Main analysis service ──────────────────────────────────────────────────
analysis:
build:
context: .
dockerfile: Dockerfile
target: runtime
image: research-env:1.0.0
container_name: research-analysis
volumes:
- ./data:/workspace/data:ro # read-only data
- ./results:/workspace/results # write results
- ./src:/workspace/src # live code update (dev)
- ./config:/workspace/config:ro
environment:
- PYTHONUNBUFFERED=1
- DATA_DIR=/workspace/data
- RESULTS_DIR=/workspace/results
command: python src/main_analysis.py
networks:
- research-net
depends_on:
- database
# ── PostgreSQL for intermediate results ───────────────────────────────────
database:
image: postgres:15-alpine
container_name: research-db
environment:
POSTGRES_DB: research
POSTGRES_USER: researcher
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
volumes:
- pgdata:/var/lib/postgresql/data
- ./sql/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
networks:
- research-net
secrets:
- db_password
# ── Jupyter Lab for interactive exploration ────────────────────────────────
jupyter:
build:
context: .
dockerfile: Dockerfile
image: research-env:1.0.0
container_name: research-jupyter
ports:
- "8888:8888"
volumes:
- .:/workspace
command: >
jupyter lab
--ip=0.0.0.0
--port=8888
--no-browser
--NotebookApp.token=''
--NotebookApp.password=''
networks:
- research-net
# ── MLflow tracking server ─────────────────────────────────────────────────
mlflow:
image: ghcr.io/mlflow/mlflow:v2.9.2
container_name: research-mlflow
ports:
- "5000:5000"
volumes:
- mlflow-data:/mlflow
command: >
mlflow server
--backend-store-uri sqlite:///mlflow/mlflow.db
--default-artifact-root /mlflow/artifacts
--host 0.0.0.0
--port 5000
networks:
- research-net
networks:
research-net:
driver: bridge
volumes:
pgdata:
mlflow-data:
secrets:
db_password:
file: ./secrets/db_password.txt
# Start all services
docker compose up -d
# Run analysis only
docker compose run --rm analysis
# View logs
docker compose logs -f analysis
# Stop and clean up
docker compose down -v
Step 3: Singularity for HPC Deployment
# Singularity definition file: research.def
# File: research.def
Bootstrap: docker
From: python:3.11.5-slim
%labels
Author research@university.edu
Version 1.0.0
Description Research analysis environment
%environment
export PATH=/opt/conda/bin:$PATH
export PYTHONUNBUFFERED=1
export OMP_NUM_THREADS=4
%post
apt-get update && apt-get install -y --no-install-recommends \
gcc g++ libopenblas-dev curl && \
rm -rf /var/lib/apt/lists/*
pip install --no-cache-dir \
numpy==1.24.4 \
pandas==2.0.3 \
scipy==1.11.4 \
scikit-learn==1.3.2 \
statsmodels==0.14.1 \
matplotlib==3.7.3
# Create workspace directory
mkdir -p /workspace /results /scratch
%runscript
echo "Research environment v1.0.0"
exec python "$@"
%test
python -c "import numpy, pandas, scipy, sklearn; print('All imports OK')"
# Build Singularity image (requires root or fakeroot)
singularity build research.sif research.def
# On HPC: convert from Docker Hub
singularity pull research.sif docker://username/research-env:1.0.0
# Run on HPC (SLURM)
singularity exec research.sif python analysis.py
# Run with GPU support
singularity exec --nv research.sif python train_model.py
# Bind mount data
singularity exec \
--bind /scratch/data:/data:ro \
--bind /scratch/results:/results \
research.sif python analysis.py
# SLURM job script: submit_job.sh
#!/bin/bash
#SBATCH --job-name=research_analysis
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=16
#SBATCH --mem=64G
#SBATCH --time=24:00:00
#SBATCH --partition=standard
#SBATCH --output=logs/job_%j.out
#SBATCH --error=logs/job_%j.err
module load singularity/3.9
singularity exec \
--bind $SCRATCH/data:/data:ro \
--bind $SCRATCH/results:/results \
--bind $SLURM_SUBMIT_DIR/src:/src:ro \
research.sif \
python /src/analysis.py \
--n-jobs ${SLURM_NTASKS} \
--output /results/output_${SLURM_JOB_ID}
Advanced Usage
Python Docker SDK for Workflow Automation
import docker
import os
import tarfile
import io
import json
from pathlib import Path
def run_analysis_container(
image="research-env:1.0.0",
script="analysis.py",
data_dir="./data",
results_dir="./results",
env_vars=None,
):
"""Run an analysis in a Docker container.
Args:
image: Docker image name:tag
script: Script to run inside container
data_dir: Local data directory to mount (read-only)
results_dir: Local results directory to mount (write)
env_vars: Additional environment variables
Returns:
dict with exit_code, logs, and output_path
"""
client = docker.from_env()
Path(results_dir).mkdir(parents=True, exist_ok=True)
environment = {"PYTHONUNBUFFERED": "1"}
if env_vars:
environment.update(env_vars)
volumes = {
os.path.abspath(data_dir): {
"bind": "/workspace/data", "mode": "ro"
},
os.path.abspath(results_dir): {
"bind": "/workspace/results", "mode": "rw"
},
}
container = client.containers.run(
image=image,
command=f"python /workspace/src/{script}",
volumes=volumes,
environment=environment,
detach=True,
remove=False, # keep for log retrieval
)
# Wait for completion
exit_code = container.wait()["StatusCode"]
logs = container.logs(stdout=True, stderr=True).decode("utf-8")
container.remove()
return {
"exit_code": exit_code,
"logs": logs,
"success": exit_code == 0,
"results_dir": results_dir,
}
# Demonstrate (without actually running Docker)
print("Docker container automation configured")
print("Usage: run_analysis_container('research-env:1.0.0', 'my_analysis.py')")
Multi-Stage GPU Build
# File: Dockerfile.gpu
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04 AS gpu-builder
RUN apt-get update && apt-get install -y --no-install-recommends \
python3.11 python3.11-dev python3-pip \
&& rm -rf /var/lib/apt/lists/*
RUN python3.11 -m pip install --upgrade pip && \
pip install --no-cache-dir \
torch==2.1.0+cu121 \
torchvision==0.16.0+cu121 \
--index-url https://download.pytorch.org/whl/cu121
COPY requirements-gpu.txt .
RUN pip install --no-cache-dir -r requirements-gpu.txt
# Runtime stage
FROM nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04 AS gpu-runtime
COPY --from=gpu-builder /usr/local/lib/python3.11 /usr/local/lib/python3.11
COPY --from=gpu-builder /usr/local/bin/python* /usr/local/bin/
RUN useradd -m -u 1000 researcher
USER researcher
WORKDIR /workspace
CMD ["python3.11", "-c", "import torch; print(f'CUDA: {torch.cuda.is_available()}')"]
Environment Freeze Script
import subprocess
import json
import platform
import sys
from datetime import datetime
def freeze_environment(output_file="environment_snapshot.json"):
"""Capture full environment state for reproducibility.
Records Python version, installed packages, system info, and git state.
"""
snapshot = {
"timestamp": datetime.now().isoformat(),
"python_version": sys.version,
"platform": platform.platform(),
"architecture": platform.architecture()[0],
}
# Installed packages
result = subprocess.run(
[sys.executable, "-m", "pip", "list", "--format=json"],
capture_output=True, text=True
)
if result.returncode == 0:
snapshot["pip_packages"] = json.loads(result.stdout)
# Git commit hash
git_result = subprocess.run(
["git", "rev-parse", "HEAD"],
capture_output=True, text=True
)
if git_result.returncode == 0:
snapshot["git_commit"] = git_result.stdout.strip()
git_status = subprocess.run(
["git", "status", "--porcelain"],
capture_output=True, text=True
)
snapshot["git_dirty"] = len(git_status.stdout.strip()) > 0
with open(output_file, "w") as f:
json.dump(snapshot, f, indent=2)
print(f"Environment snapshot saved to {output_file}")
print(f" Python: {snapshot['python_version'].split()[0]}")
print(f" Packages: {len(snapshot.get('pip_packages', []))}")
print(f" Git commit: {snapshot.get('git_commit', 'unknown')[:8]}")
return snapshot
snapshot = freeze_environment()
Troubleshooting
| Problem | Cause | Fix |
|---|---|---|
permission denied: /var/run/docker.sock | User not in docker group | sudo usermod -aG docker $USER && newgrp docker |
| Container exits immediately | Script error or CMD syntax | Run with bash to debug: docker run --rm -it image bash |
| Large image size | Including data or build artifacts | Use .dockerignore; multi-stage build |
| Volume mount fails (Windows) | Path format | Use //c/Users/... or Docker Desktop volume GUI |
| Singularity on HPC: no root | Requires fakeroot or admin | Use --fakeroot flag or pull pre-built Docker image |
| GPU not visible in container | Missing NVIDIA runtime | Install nvidia-container-toolkit; run with --gpus all |
External Resources
- Docker documentation
- Singularity/Apptainer user guide
- Reproducible research with containers (The Turing Way)
- NVIDIA Container Toolkit
- Docker Python SDK
Examples
Example 1: Research Compendium with Make
# File: Makefile
.PHONY: build run clean push
IMAGE := research-env
TAG := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")
build:
docker build -t $(IMAGE):$(TAG) -t $(IMAGE):latest .
docker inspect $(IMAGE):$(TAG) | python -c \
"import json,sys; img=json.load(sys.stdin)[0]; print(f'Size: {img[\"Size\"]/1e6:.1f} MB')"
run:
docker compose up -d
@echo "Jupyter at http://localhost:8888"
analysis:
docker compose run --rm analysis python src/main_analysis.py
clean:
docker compose down -v
docker image rm $(IMAGE):$(TAG) 2>/dev/null || true
push:
docker push $(IMAGE):$(TAG)
docker push $(IMAGE):latest
Example 2: Verify Container Reproducibility
import subprocess
import hashlib
import json
def verify_analysis_reproducibility(script, data_hash, n_runs=3):
"""Run a script multiple times and verify output hashes match.
Args:
script: path to analysis script
data_hash: SHA256 hash of input data
n_runs: number of repeated runs
Returns:
dict with consistency information
"""
output_hashes = []
for run in range(n_runs):
# In practice: docker run --rm image python script
# Here: simulate by running locally
result = subprocess.run(
["python", script, "--seed", "42"],
capture_output=True, text=True
)
output_hash = hashlib.sha256(result.stdout.encode()).hexdigest()
output_hashes.append(output_hash)
print(f"Run {run+1}: {output_hash[:16]}...")
all_same = len(set(output_hashes)) == 1
print(f"\nReproducibility: {'PASS' if all_same else 'FAIL'}")
print(f"Input data hash: {data_hash[:16]}...")
return {"reproducible": all_same, "hashes": output_hashes}
示例提示词
帮我把这套分析环境打包成Docker容器,方便在HPC上复现运行。
所属插件
安装该插件会一并获得同领域的 54 个 Skill / MCP。
同插件的其他科研 Skill
- 智能体资源节约守则Applies token- and context-saving habits for tool choice, PDF handling, persistent memory, and privacy checks. Use at the start of sessions with many files or PDFs, or multi-session work.
- 英文论文双语精读器把英文论文转成中英对照精读档并生成离线双语阅读器,标注要点并输出读书笔记。精读单篇英文论文、要中英对照阅读时使用。
- 研究方法比较论证Compares quantitative, qualitative, and mixed research methods against what each can and cannot answer to build a methods table. Use when justifying a method choice in a thesis.
- 复杂系统分析方法Analyzes complex systems with power-law fitting, Hurst exponent, fractal dimension, agent-based modeling, and entropy measures. Use for nonlinear or complex-system data.
- DVC数据版本管理Sets up DVC for dataset and pipeline versioning, remote storage, and experiment comparison in reproducible ML research workflows. Use when tracking large datasets or ML experiment runs.
- 期刊级统计图表绘制Creates publication-quality multi-panel figures with matplotlib, seaborn, or ggplot2 using colorblind-safe palettes and journal export settings. Use when preparing figures for a paper.
- 综述模块拆解规划Breaks a refined literature-review question into 3-5 non-overlapping modules to build a reading map, outline, or search plan. Use when structuring a literature review.
- 学位论文答辩教练模拟学位口试委员提问并提供承认回应证据三段式答辩框架与模拟问答。准备论文口试或研讨会问答环节时使用。
- 实验设计与样本量规划Designs rigorous experiments covering sample-size calculation, randomization strategy, pre-registration, and A/B test duration estimation. Use when planning an experiment or A/B test.
- 管理实验设计架构为管理与商学情境实验规划组间组内设计、对抗平衡与操弄检核,并产出设计书与情境材料模板。做商学实验设计时使用。
- 科研经费申请撰写Structures research grant proposals including NSF/NIH Specific Aims, budget justification, ERC narrative sections, and biosketch formatting. Use when drafting a funding application.
- 人类可读文档撰写规范Guides writing human-facing docs like README, PLAN, survey notes, and reviewer-response drafts with a clear two-tier structure. Use for text a human collaborator will read.
- 知识图谱与SPARQL检索Builds knowledge graphs and runs SPARQL queries over RDF and Wikidata for entity linking in research and library data. Use when modeling research entities as a linked graph.
- LaTeX论文写作与排版Writes, compiles, and submits LaTeX papers covering IMRaD structure, key packages, bibliography management, and arXiv preparation. Use when authoring or debugging a LaTeX manuscript.
- 文献比较矩阵构建器把一批PDF整理成含撤稿查核与APA7书目的文献资料库,并生成横向比较矩阵表。整理综述前要比较文献时使用。
- 跨库学术文献检索Searches OpenAlex, Semantic Scholar, and arXiv for academic literature with deduplication, citation snowballing, and BibTeX export. Use when collecting papers for a review or topic.
- 文献调研语料库维护Finds, verifies, and maintains a literature corpus from arXiv, OpenAlex, Crossref, and related databases with BibTeX entries and per-paper notes. Use when sourcing references for a paper.
- 文献综述研究缺口定位Converts a literature-review synthesis into a concrete, defensible research gap and future-research direction. Use when writing a conclusion or thesis contribution section.
- 文献矛盾结论梳理Analyzes conflicting findings across papers and explains disagreement by sample, method, or context differences for a stronger synthesis. Use when review sources disagree.
- 效应量汇总元分析Pools effect sizes across studies with fixed and random-effects models, heterogeneity tests, forest and funnel plots, and subgroup analysis. Use to synthesize quantitative study results.
- 缺失数据处理Diagnoses missing-data mechanisms such as MCAR, MAR, or MNAR and applies multiple imputation or deletion with sensitivity analysis. Use when a dataset has missing values.
- 混合方法研究设计Designs mixed-methods studies integrating qualitative coding, triangulation, content analysis, and QCA with inter-rater reliability checks. Use when combining qual and quant data.
- 科研机器学习流程Applies machine learning to research questions with scikit-learn pipelines, AutoML, SHAP explainability, and nested cross-validation. Use when building a predictive model for a study.
- 开放数据存储与FAIRDeposits and documents research data via Zenodo or OSF with FAIR assessment, data management plans, and DataCite metadata. Use when publishing or sharing a research dataset.
常见问题
研究环境容器化部署能做什么?
Builds Docker and Singularity containers for reproducible computation and HPC deployment of research environments. Use when packaging an analysis pipeline for reuse or HPC。
如何安装研究环境容器化部署?
在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「研究环境容器化部署」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。
研究环境容器化部署需要哪些权限?
该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。
研究环境容器化部署经过官方审核吗?
是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。