jd-worker — Library Reference

? Overview

The jd-worker package provides two things:

1. A CLI tool (jd_worker_cli) — the worker process you run in your terminal. It polls the job server every ~3 minutes via POST /worker/heartbeat (liveness, dashboard control, and job assignment when idle), heartbeats every 57 s while a job runs, invokes your script once per job, and reports results.
2. A Python library (import jd) — a small set of helper functions your script imports to resolve correct file paths and upload results to the central server.

The library functions are designed to be zero-configuration. They read the context (experiment name, job ID, server URL) from environment variables that jd_worker_cli sets automatically before calling your script. You never need to pass these values manually.

1 Installation

pip install jd-worker

# Upgrade to the latest version at any time
pip install --upgrade jd-worker

Requires Python 3.8 or later. Install inside a virtual environment (see the Getting Started guide for instructions).

After installation, verify both the CLI and the library are available:

jd_worker_cli help
export JD_API_KEY=jd_…   # required for interactive mode
jd_worker_cli          # interactive shell (API key verified with Hub)
python -c "import jd; print(jd.__version__)"

2 jd_worker_cli — The Worker Command

jd_worker_cli is the main executable installed by the package. In its default mode it runs as a long-lived background process that:

1. Connects to the Hub to obtain a short-lived worker token and the server URL for your experiment (Hub mode).
2. Registers itself in a local SQLite registry so you can list, stop, and manage workers without tmux or process hunting.
3. Polls the server for the next PENDING job.
4. Sets environment variables describing the job and calls your entry script as a subprocess.
5. Sends POST /worker/heartbeat every 57 seconds while the script runs.
6. Marks the job DONE or ABORTED depending on the exit code, then goes back to step 3.
7. Keeps running when the queue is empty — probing every 3 minutes (180 s idle heartbeat) for new jobs. Use once=true to exit after one job or when no job is available.

On every launch (except help), the CLI first scans the local registry SQLite databases, removes worker rows whose PIDs are no longer running, releases orphaned instance names, and deletes empty experiment cache directories. Large machines (50+ registered workers) may show a short progress bar with ETA during this step; later commands in the same process skip redundant work.

Arguments use key=value tokens. Subcommands such as worker-list or exp-status are bare words mixed with the same tokens. Run jd_worker_cli help for the full embedded reference.

Starting workers

The run command (no subcommand) starts one or more workers:

export JD_API_KEY=jd_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

jd_worker_cli expId=digits-tune entry_script=train.py

Required to start:

Launch arguments

Arguments can be passed as key=value tokens or via the corresponding environment variable.

Background workers (default)

By default, workers detach to the background — no tmux required. The launcher prints each worker’s worker_id and PID, then returns immediately. Process metadata is stored in:

~/.jd_cache/.cache/<expId>/workers.db
# or <JD_CACHE_PATH>/.cache/<expId>/workers.db

Each worker gets a worker_id like gpunode_egg_0 ({host}_{instance}_{slot}). The instance segment is a short random object name (e.g. egg, table, max 6 letters) allocated from a pool in workers.db. Legacy workers may still use 6-character alphanumeric tokens. Standalone launches always use slot 0; num_workers=N uses slots 0 … N-1. The same id appears in worker-list, on the server as requested_by, and in log filenames.

# Start four background workers (returns immediately)
jd_worker_cli expId=digits-tune entry_script=train.py num_workers=4

# List workers for one experiment
jd_worker_cli expId=digits-tune worker-list

# List all experiments with worker counts on this machine
jd_worker_cli exp-list

# Stop one worker or all
jd_worker_cli expId=digits-tune stop gpunode_egg_0
jd_worker_cli expId=digits-tune stop all

Use foreground=true when debugging — logs stream to the terminal and the process stays attached until you press Ctrl+C.

Interactive mode

Run jd_worker_cli with no arguments (or interactive / -i) for a mysql-style REPL. This is useful on shared clusters where you manage workers over a long SSH session without retyping expId= every time.

jd_worker_cli
jd_worker_cli -i expId=digits-tune

jd_worker_cli 1.15.0 — interactive mode
Type help for commands, exit or Ctrl-D to quit.
jd> use digits-tune
Using experiment 'digits-tune'.
jd[digits-tune]> worker-list
jd[digits-tune]> exp-status
jd[digits-tune]> entry_script=train.py num_workers=4
jd[digits-tune]> stop all
jd[digits-tune]> exit

Once use is set, omit expId= on later lines. Trailing semicolons are optional. Command history is saved to ~/.jd_cache/.cache/jd_worker_history (under JD_CACHE_PATH when set).

Management commands

These subcommands manage workers already running on the machine, query the server, or perform housekeeping. Most require expId=<name> unless you are in interactive mode with use set.

jd_worker_cli expId=digits-tune exp-status
jd_worker_cli expId=digits-tune worker-status gpunode_egg_0
jd_worker_cli expId=digits-tune worker-logs gpunode_egg_0 lines=100 follow=true
jd_worker_cli expId=digits-tune server-info
jd_worker_cli expId=digits-tune scale num_workers=8
jd_worker_cli expId=digits-tune drain
jd_worker_cli expId=digits-tune stop job=42
jd_worker_cli prune

Heartbeat protocol (POST /worker/heartbeat)

Workers registered with jd-worker 1.15.0+ use POST /worker/heartbeat for liveness, job assignment, and dashboard control. Every 180 seconds when idle (and every 57 seconds while a job runs), the worker sends:

• worker_id, host, machine_type
• reported_status — idle or busy
• current_job_id — set while running a job
• applied_version — ack of the last dashboard command applied
• system_metrics — CPU/RAM snapshot

The server responds with:

• desired_state — run, pause, drain, or stop
• desired_version / applied_version — command versioning
• job — optional job payload when idle and desired_state=run
• heartbeat_interval — seconds until the next heartbeat (180 idle, 57 busy)

While a job runs, a background heartbeat thread keeps the worker and job rows fresh and picks up dashboard commands. Stop and drain take effect after the current job completes (or immediately when idle for stop).

Dashboard worker control

The server dashboard shows live idle and busy worker counts in the Jobs sidebar. Open the Workers Management nav tab for the full UI: active vs disabled workers, host/instance/slot filters, per-worker history, and metrics. Open Details on any row to view history (newest first) and system metrics like the job detail modal.

• Filter workers by host, instance, and slot
• Queue Resume, Pause, Drain, or Stop on the Active Workers sub-tab
• Review Disabled / Stopped workers (read-only) after shutdown or lost heartbeat
• Cancel a queued command before the worker’s next idle heartbeat (~3 min / 180 s)

Dashboard commands set desired state on the server — they do not send signals directly to worker processes. Precedence is stop > drain > pause > run. Local CLI commands (stop, drain) still work on the machine where the worker runs; dashboard control is for remote management without SSH.

Examples

Standard Hub mode:

export JD_API_KEY=jd_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

jd_worker_cli expId=digits-tune entry_script=train.py machine_type=cpu

Four background workers on one machine:

jd_worker_cli expId=digits-tune entry_script=train.py \
              num_workers=4 machine_type=gpu

Smoke-test one job in the foreground:

jd_worker_cli expId=digits-tune entry_script=train.py \
              foreground=true once=true

Standalone server (no Hub):

jd_worker_cli expId=digits-tune entry_script=train.py \
              server=192.168.1.10 port=8000

Custom data root on scratch disk (HPC):

# Shared storage for large job I/O
export JD_WORKSPACE_PATH=/scratch/$USER
# Node-local registry (SQLite) — one .jd_cache per compute node
export JD_CACHE_PATH="${TMPDIR:-/tmp}/.jd_cache"
mkdir -p "$JD_WORKSPACE_PATH" "$JD_CACHE_PATH"

jd_worker_cli expId=digits-tune entry_script=train.py num_workers=4

3 Path Helper Functions

These functions return pathlib.Path objects pointing to the correct directories on the worker machine. They work by reading the JD_JOB_ID and JD_EXP_ID environment variables that jd_worker_cli sets before calling your script. You only need to import and call them — no arguments required.

from jd import jd_job_dir

job_dir = jd_job_dir()
job_dir.mkdir(parents=True, exist_ok=True)

# Write a result file
(job_dir / "result.json").write_text('{"accuracy": 0.97}')

# Write a model checkpoint
import torch
torch.save(model.state_dict(), job_dir / "model.pt")

from jd import jd_exp_dir
from pathlib import Path

# Load a shared dataset stored once in the experiment directory
dataset_path = jd_exp_dir() / "dataset.csv"

if not dataset_path.exists():
    # Download it on first use
    download_dataset(dataset_path)

# Store all job data on a fast scratch disk instead of ~
export JD_WORKSPACE_PATH=/scratch/$USER

jd_worker_cli expId=digits-tune entry_script=train.py num_workers=4

from jd import jd_worker_workspace

workspace = jd_worker_workspace()
# ~/jd_data/  (default)   or   /scratch/alice/jd_data/  (if JD_WORKSPACE_PATH is set)

print(workspace)  # Path object — e.g. /home/alice/jd_data

4 File Upload

from jd import jd_job_dir, jd_upload
import json

job_dir = jd_job_dir()
job_dir.mkdir(parents=True, exist_ok=True)

# Save result locally first
result_path = job_dir / "result.json"
result_path.write_text(json.dumps({"accuracy": 0.97}, indent=2))

# Upload to the server so it's visible in the dashboard
jd_upload(result_path)

# You can upload multiple files from the same job
jd_upload(job_dir / "confusion_matrix.png")
jd_upload(job_dir / "model.pt")

5 Checkpoints

Two functions let your entry script save intermediate training state to the server and resume from it on restart:

from jd import jd_update_checkpoint, jd_get_last_checkpoint

# Resume if a checkpoint exists (returns None on first run)
ckpt        = jd_get_last_checkpoint()
start_epoch = ckpt["epoch"] + 1 if ckpt else 0
if ckpt:
    model.load_state_dict(ckpt["model"])

# Save a checkpoint every 5 epochs
for epoch in range(start_epoch, args.epochs):
    train_one_epoch(model, optimizer)
    if (epoch + 1) % 5 == 0:
        jd_update_checkpoint({"epoch": epoch, "model": model.state_dict()})

For the full guide — including patterns for saving the best model, non-PyTorch objects, size limits, and a comparison with local file saves — see the dedicated Checkpointing Guide.

6 Environment Variables

jd_worker_cli sets these environment variables in the subprocess that runs your script. You can read them directly with os.environ when you need values that the path helpers do not expose.

Example — read job ID and experiment name directly:

import os

job_id  = int(os.environ["JD_JOB_ID"])
exp_id  = os.environ["JD_EXP_ID"]
print(f"Running job {job_id} for experiment '{exp_id}'")

Additionally, the following variables from the worker's own environment are inherited by your script and can also be used to control jd_worker_cli itself:

7 Worker Logs

Each worker process writes a log file capturing job fetches, status updates, ping activity, and the combined stdout/stderr from your entry script. Background workers log to file only; use foreground=true to mirror logs to the terminal.

Default log location:

~/jd_data/<exp_id>/jd_worker_logs/jd_worker_<worker_id>.log

worker_id is formatted as {host}_{instance}_{slot} (e.g. gpunode_egg_0). Override the directory with log_dir=<path> or JD_LOG_DIR.

View logs via the CLI (uses the path stored in the worker registry):

# Last 50 lines
jd_worker_cli expId=digits-tune worker-logs gpunode_egg_0

# Last 200 lines, then follow live
jd_worker_cli expId=digits-tune worker-logs gpunode_egg_0 lines=200 follow=true

Or tail the file directly:

tail -f ~/jd_data/digits-tune/jd_worker_logs/jd_worker_*.log

Use worker-status <worker-id> to see the exact log path for a registered background worker.

8 Effective Usage Patterns

Pattern 1 — Minimal script (result only)

For quick experiments where you only need a single JSON result and no file upload:

import argparse, json
from jd import jd_job_dir

parser = argparse.ArgumentParser()
parser.add_argument("--lr", type=float)
parser.add_argument("--epochs", type=int)
args = parser.parse_args()

# ... train ...
accuracy = 0.95

job_dir = jd_job_dir()
job_dir.mkdir(parents=True, exist_ok=True)
(job_dir / "result.json").write_text(json.dumps({
    "lr": args.lr, "epochs": args.epochs, "accuracy": accuracy
}))

Pattern 2 — Ephemeral nodes (Slurm / cloud) with upload

When nodes lose their local storage after the job ends, upload everything important before the script exits:

from jd import jd_job_dir, jd_upload

job_dir = jd_job_dir()
job_dir.mkdir(parents=True, exist_ok=True)

# ... train and save outputs ...

# Upload all files in the job directory
for f in job_dir.iterdir():
    if f.is_file():
        jd_upload(f)

Pattern 3 — Shared dataset across jobs

Download a dataset once per machine to the experiment directory, then load it for every job without re-downloading:

from jd import jd_exp_dir
import filelock

dataset_path = jd_exp_dir() / "data.csv"
lock_path    = jd_exp_dir() / "data.csv.lock"

# Thread-safe: only the first worker downloads; others wait and reuse
with filelock.FileLock(str(lock_path)):
    if not dataset_path.exists():
        download_dataset(dataset_path)

df = pd.read_csv(dataset_path)

Pattern 4 — Resumable training with checkpoints

For long-running jobs on preemptible hardware, use jd_update_checkpoint() periodically and jd_get_last_checkpoint() at startup to resume automatically from the last saved epoch. See the Checkpointing Guide for complete examples including best-model saving, non-PyTorch objects, and a full resume pattern.