科研 Skill
HPC作业运行诊断
hpc-runtime-doctor
Diagnoses HPC runtime and scheduler problems for materials simulations: MPI/OpenMP/GPU layout, modules, scratch paths, walltime. Use when a cluster job fails, hangs, or underperforms.
SKILL.md
HPC Runtime Doctor
Goal
Turn cluster symptoms into a resource-layout diagnosis, environment checklist, and safe retry plan.
Requirements
- Python 3.10+
- No external dependencies
- Works on Linux, macOS, and Windows
Inputs to Gather
| Input | Description | Example |
|---|---|---|
| Scheduler | SLURM, PBS, LSF, local | slurm |
| Nodes/tasks/threads | Runtime layout | 2 nodes, 128 tasks, 2 threads |
| GPUs | Total (whole-job) GPUs via --gpus, or per node via --gpus-per-node | --gpus 4 or --gpus-per-node 1 |
| Symptoms | Observed failure | oom,killed,slow-gpu |
| MPI/OpenMP/GPU use | Parallel modes | mpi+openmp+gpu |
| Walltime | Requested time | 12:00:00 |
| Scratch | Whether scratch is used | true |
Decision Guidance
- Check resource layout before changing physics settings.
- Confirm module/compiler/MPI/CUDA consistency before debugging solver behavior.
- Treat missing restart files and scratch cleanup as workflow failures, not physics failures.
- For GPU jobs, confirm the executable was built with the requested accelerator backend.
Script Outputs
scripts/hpc_runtime_doctor.py emits:
resource_layout(includestasks_per_node,total_cpus, totalgpus, andgpus_per_node)diagnosesenvironment_checksretry_planscheduler_noteswarnings(layout flags such as ranks-per-GPU oversubscription, OpenMP/thread mismatch, and uneven task placement)
In default (non-JSON) mode the script also prints the resource-layout summary, any
warnings, environment checks, and retry plan, so the most actionable items are never hidden.
Workflow
--gpus is the total (whole-job) GPU count. Use --gpus-per-node (SLURM
--gres=gpu:N semantics) when you know the per-node allocation; total GPUs are then
gpus_per_node * nodes and it overrides --gpus.
python3 skills/hpc-deployment/hpc-runtime-doctor/scripts/hpc_runtime_doctor.py \
--scheduler slurm \
--nodes 2 \
--tasks 128 \
--cpus-per-task 2 \
--gpus 4 \
--symptoms oom,slow-gpu \
--uses-mpi \
--uses-openmp \
--uses-gpu \
--json
The example above shares 128 ranks across 4 GPUs (32 ranks/GPU), so the
warnings list surfaces Many MPI ranks per GPU (32.0 ranks/GPU) may reduce GPU efficiency. The ranks-per-GPU check uses total ranks over total GPUs, so it fires
correctly on multi-node jobs (the threshold is 16 ranks/GPU).
Error Handling
Invalid resource counts stop with exit code 2. Unknown symptoms are preserved as custom items for human review.
Limitations
This skill does not query a live scheduler. It diagnoses from the submitted layout and symptoms.
Verification checklist
- Recorded the script's
resource_layoutblock and confirmedtasks_per_nodeis an integer (no fractional value) andtotal_cpusequalstasks * cpus_per_task; iftasks_per_nodeis fractional, the uneven-placement warning was triaged before retrying. - For GPU jobs, recorded the resolved total
gpus(andgpus_per_nodewhen set) and computed ranks/GPU =tasks / gpus, confirming it is at or below the 16 ranks/GPU threshold or that the resultingMany MPI ranks per GPUwarning was deliberately accepted. - Reviewed every entry in the
warningslist (OpenMP-with-cpus_per_task=1, GPU-requested-but-zero-GPUs,tasks < nodes, uneven placement, scratch-for-heavy-I/O) and resolved or justified each one rather than ignoring it. - Completed the
environment_checksitems as real artifacts: captured the module list, executable path/version, MPI launcher-vs-library match, accelerator build flags (CUDA/Kokkos/OpenMP), and scheduler stdout/stderr. - Mapped each observed symptom to a
diagnosesentry and verified no symptom landed in thecustomcategory unaddressed (everycustomitem had stderr/stdout/module list/command line collected for human review). - Followed the
retry_plan: reran the smallest reproducing case, changed exactly one resource variable, enabled restart/checkpoint, and saved the scheduler script plus environment snapshot alongside the results.
Common pitfalls & rationalizations
| Tempting shortcut | Why it's wrong / what to do |
|---|---|
| "It ran without crashing, so the layout is fine." | Run completion is not correctness. Review the warnings list and resource_layout -- oversubscription, uneven placement, or an idle GPU can silently slow or corrupt results without a crash. |
| "Per-node ranks fit the GPUs, so there's no oversubscription." | Oversubscription is total ranks over total GPUs, not per-node. The script computes tasks / gpus; a multi-node job can hide a high ranks/GPU value that only the unit-consistent check exposes. |
"I passed --gpus, so per-node GPU count doesn't matter." | --gpus is the whole-job total. If the cluster allocates per node, use --gpus-per-node (SLURM --gres=gpu:N); it overrides --gpus and total becomes gpus_per_node * nodes. Mixing them up misreports ranks/GPU. |
| "The job was killed, so it's a physics/solver bug." | killed/oom/timeout are scheduler and resource categories, not physics. Check walltime, memory limits, and preemption from stdout/stderr before touching simulation parameters. |
| "An unknown symptom isn't in the rules, so I can skip it." | Unknown symptoms become custom diagnoses, not no-ops. Collect scheduler stderr/stdout, the module list, and the command line for human review -- silence is not a clean bill of health. |
| "Just change ranks, threads, and the build together to fix it faster." | Changing multiple variables at once makes the failure undiagnosable. The retry_plan mandates one variable at a time on the smallest reproducing case. |
Security
Input Validation
- Inputs are scalar CLI values and booleans only; there is no free-form code path.
- Resource counts (
--nodes,--tasks,--cpus-per-task,--gpus,--gpus-per-node) are validated as integers (booleans rejected), required to be non-negative and finite, and capped at 1,000,000.--nodes,--tasks, and--cpus-per-taskmust additionally be at least 1. Out-of-range, non-integer, or zero values exit with code 2. - The
--symptomsstring is capped at 64 comma-separated entries of at most 64 characters each;--walltimeis capped at 32 characters. Oversized input exits with code 2. - Symptoms are split, trimmed, and lower-cased. Unknown symptoms are not rejected: they are
preserved as
customdiagnoses for human review. --scheduleris accepted as a free-form string and is not checked against an allowlist; it is only echoed back in the resource layout.
File Access
- The script reads and writes no files. All I/O is CLI args in and stdout out
(indented JSON with
--json, otherwise a human-readable summary); errors go to stderr. - Because no paths are accepted or opened, there is no filesystem traversal surface and no path-sandboxing concern.
Tool Restrictions
allowed-toolsisRead, Bash, Write, Grep, Glob.Bashis used only to run the bundledscripts/hpc_runtime_doctor.py.Read,Grep, andGlobare used to inspect the skill's own files and any logs or submission scripts the user points at;Writeis used to record diagnosis notes or a retry plan when asked.
Safety Measures
- No
eval,exec,os.system, orsubprocess; the script does not launch a scheduler or any external process and does not inspect environment variables. - Argument parsing is handled by
argparse, and machine-readable output is emitted as JSON. - DoS exposure is bounded by the resource-count cap (1,000,000), the symptom caps (64 entries x 64 characters), and the walltime cap (32 characters).
References
- See
references/hpc_runtime_patterns.mdfor scheduler and runtime diagnosis patterns.
Version History
- 1.1.3: Added a Verification checklist (evidence-based items tied to
resource_layout,warnings, ranks/GPU,environment_checks,diagnoses, and theretry_plan) and a Common pitfalls & rationalizations table. - 1.1.1: Discriminating evals -- each case now pins the script's specific output
(exact ranks-per-GPU warning, diagnosis categories, resource-layout fields) via
deterministic
script_checks. - 1.1.0: Unit-consistent ranks-per-GPU warning (total ranks / total GPUs), new
--gpus-per-nodeargument, integertasks_per_nodewith an uneven-placement warning, full human-readable (non-JSON) output, and input caps for resource counts, symptoms, and walltime. - 1.0.0: Initial HPC runtime diagnosis skill.
示例提示词
我的LAMMPS作业在集群上跑崩了,帮我诊断MPI配置和资源分配问题。
所属插件
安装该插件会一并获得同领域的 23 个 Skill / MCP。
同插件的其他科研 Skill
- 制造解法验证与基准测试规划Plans verification and validation campaigns using manufactured solutions, canonical benchmarks. Use when an agent must prove a solver is correct before trusting results.
- 数值解空间时间收敛性分析Performs spatial/temporal convergence analysis for solution verification: observed order, Richardson extrapolation. Use when checking that a simulation result is grid/timestep independent.
- PDE数值微分格式选择Selects and applies numerical differentiation schemes for PDE/ODE discretization: finite-difference stencils. Use when choosing a spatial discretization scheme.
- 仿真结果FAIR归档打包Creates FAIR-minded reproducibility bundles for simulations: file inventories, hashes, units, engine versions. Use before publishing or archiving a simulation dataset.
- 线性方程组求解器选型Selects and configures linear solvers for Ax=b systems: direct (LU, Cholesky) versus iterative (CG, GMRES, BiCGSTAB) methods, sparsity. Use when choosing a linear solver for a simulation.
- 分子动力学分析方案规划Plans molecular dynamics post-processing for materials simulations: RDF, MSD/diffusion, VACF/VDOS, coordination numbers. Use when deciding which MD analysis to run and in what order.
- 数值模拟网格生成规划Plans and evaluates mesh generation for numerical simulations: grid resolution from physics scales, aspect ratio/skewness checks. Use when designing or auditing a simulation mesh.
- 非线性方程组求解器选型Selects and configures nonlinear solvers for root-finding, optimization, and least-squares problems: Newton, quasi-Newton, Broyden, Anderson acceleration. Use when choosing a nonlinear solver.
- 时间积分方法选择Selects and configures time-integration methods for ODE/PDE simulations: explicit Runge-Kutta, BDF, Rosenbrock. Use when choosing a time-stepping scheme for a solver.
- 偏微分方程数值稳定性分析Analyzes numerical stability for time-dependent PDE simulations: CFL/Fourier criteria, von Neumann analysis. Use when a simulation blows up or to check a scheme’s stability limits.
- 材料科学本体浏览查询Parses, navigates, and queries materials science ontology structures: class hierarchies, property domain/range lookup. Use when exploring an unfamiliar materials ontology.
- 材料术语本体映射Maps materials science terms, crystal structures, and sample descriptions to standardized ontology classes with confidence scores. Use when annotating data for ontology interoperability.
- 本体标注一致性校验Validates material sample annotations against ontology constraints: class/property existence, domain/range consistency. Use before publishing ontology-annotated materials data.
- 仿真参数优化与敏感性分析Explores and optimizes simulation parameters via design of experiments, sensitivity analysis, and optimizer selection: Latin Hypercube, quasi-random. Use when calibrating simulation parameters.
- 仿真计算性能剖析Identifies computational bottlenecks, parallel scaling, and memory requirements for materials simulations: parses timing logs for dominant phases. Use when a simulation is slow or scales poorly.
- 仿真输出数据后处理Extracts, analyzes, and summarizes simulation output data: spatial fields at timesteps, time-series trends, steady-state detection. Use when turning raw simulation output into summary statistics.
- 多代码仿真故障分诊Provides first-response triage for cross-code LAMMPS/VASP/QE/MOOSE simulation failures: classifies nonconvergence, NaN/Inf. Use when a simulation crashes and the cause is unclear.
- 多任务仿真批量编排Orchestrates multi-simulation campaigns: generates parameter sweeps (grid, linspace, Latin Hypercube), tracks batch job status. Use when running and managing many related simulation jobs.
- 仿真三阶段有效性校验Validates simulations across three stages: pre-flight config checks, runtime log monitoring for NaN/Inf and residual growth. Use to catch simulation errors before, during, and after a run.
- SLURM作业脚本生成Generates correct SLURM sbatch job scripts and sanity-checks HPC resource requests: nodes, MPI/OpenMP layout, memory, GPUs, walltime. Use when writing or debugging a SLURM submission script.
- 瞬态仿真时间步策略Plans and controls time-step policies for transient simulations: CFL and physics-based stability limits, adaptive stepping. Use when setting up a time-stepping strategy for a solver.
- 计算材料工作流引擎选型Maps computational materials tasks onto workflow engines (atomate2, jobflow, AiiDA, pyiron) or a one-off script. Use when deciding how to structure a reproducible campaign or DAG.
常见问题
HPC作业运行诊断能做什么?
Diagnoses HPC runtime and scheduler problems for materials simulations: MPI/OpenMP/GPU layout, modules, scratch paths, walltime. Use when a cluster job fails, hangs, or underperforms。
如何安装HPC作业运行诊断?
在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「HPC作业运行诊断」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。
HPC作业运行诊断需要哪些权限?
该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。
HPC作业运行诊断经过官方审核吗?
是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。