# Dask Jobqueue On Slurm

Use this skill when a Python workload should use Dask workers submitted through
Slurm with explicit worker resources, logs, temporary directories, and scale
limits.

## Example

Review the account, partition, and Python environment placeholders, then start
with the dry-run mode:

```bash
bash examples/dask-slurm-demo.sh
```

Dry-run mode prints the Slurm worker job script generated by Dask Jobqueue
without submitting worker jobs. To run a tiny test on a Slurm test partition:

```bash
RUN_DASK=1 \
DASK_ACCOUNT=<account> \
DASK_QUEUE=<partition> \
DASK_JOBS=1 \
PYTHON_ENV=/path/to/venv \
  bash examples/dask-slurm-demo.sh
```

Keep `DASK_JOBS`, `DASK_CORES`, `DASK_MEMORY`, and `DASK_WALLTIME` small until
the site-specific worker shape has been reviewed.

## Pattern

- Prepare a Python environment with `dask`, `distributed`, and `dask-jobqueue`.
- Configure one worker job shape at a time: cores, processes, memory, walltime,
  queue, and account.
- Put worker logs and local temporary data under explicit user-owned paths.
- Run `RUN_DASK=0` first and inspect `cluster.job_script()`.
- Scale to a small number of jobs before trying adaptive or large deployments.
- Record the scheduler address, dashboard link, worker count, and result file.
- Close the client and cluster so worker jobs are cleaned up.

## Safety Notes

This skill is `medium` risk because `dask-jobqueue` can submit multiple Slurm
worker jobs from one driver process. Start in dry-run mode, use short wall
times, and avoid adaptive scaling until site policy and limits are clear.
Worker temporary directories can grow quickly, so point them at scratch or
another approved filesystem.

## Success Criteria

- Dry-run mode prints a Slurm worker job script with the expected account,
  queue, cores, memory, walltime, and log directory.
- `RUN_DASK=1` starts the requested number of worker jobs on a test partition.
- The Dask client reports workers before computation starts.
- The demonstration workload writes a JSON result report under the output
  directory.
- Worker jobs terminate when the driver exits.