# Replicating the Docker HEART experiment, step by step

This experiment adds a HEART (https://heartcortex.ai) to **the real Docker Engine**, in two ways, and tests both with real containers and real faults. Nothing is simulated.

| Way | What changes in Docker | What runs |
|---|---|---|
| **A. Built in** | One new package in Docker's source, `daemon/heart`, and one line that starts it | `dockerd-heart`: Docker's own daemon, rebuilt from source with the HEART inside |
| **B. Sidecar** | Nothing | `heart-docker-sidecar`: the same HEART code as its own small program, beside Docker's official, unchanged `dockerd` |

Both serve the same HEART API on `127.0.0.1:43278`:
- the Card (`/.well-known/heart-card.json`);
- the Pulse (`/heart/v1/pulse`);
- alerts (`/heart/v1/arrhythmias`).

Both read Docker **only through its Engine API socket**, as any client would.

Everything below is what we ran, with the exact versions. Our results are at the end. Expect very similar ones: the times depend a little on how busy the machine is.

---

## 0. What you need

| | |
|---|---|
| **A Linux machine where containers can run** | x86-64, root access, a kernel with cgroup v2 (we used Linux 6.8 on Ubuntu 22.04 and 24.04), overlay2. **Not** an unprivileged container: on RunPod, for example, `dockerd` starts but cannot start containers (`unshare: operation not permitted`). |
| **Disk and memory** | About 3 GB of free disk for the source and build, or 350 MB if you only test. About 500 MB of memory. |
| **Go 1.26.10** | The version Docker 29.9.3 is built with: https://go.dev/dl/ |
| **Python 3.10 or newer** | With `jsonschema[format] >= 4.18` and `referencing`, to check every reply against the published schemas |
| **git, curl, tar, sha256sum** | |
| **This repository** | Folder `experimentalDocker/`, plus `spec/schemas/1.0/` and `heart-rating/` (the schemas and the HEART-Rating client) |

Your own Docker, if you have one, is never touched. The tests run their own daemons, each with its own data folder, socket and process file, no bridge network and no firewall changes. Their containers have networking switched off.

---

## 1. Get Docker's source

```sh
git clone --depth 1 --branch docker-v29.9.3 https://github.com/moby/moby
cd moby && git rev-parse HEAD        # we built 70efc0e (see results/*/moby-commit.txt)
```

## 2. Add the HEART (way A only)

```sh
python3 ../experimentalDocker/apply_patch.py .
```

It makes exactly two changes, and refuses a tree it does not recognise:

1. **The package:** it copies `experimentalDocker/heart/*.go` to `daemon/heart/`. It uses only Go's standard library, so it adds no dependency.
2. **The hook:** in `daemon/command/daemon.go`, it imports the package and adds one call right after `notifyReady()`, once every Engine API listener is serving:

   ```go
   heart.Start(ctx, cli.Config.Hosts, cli.Config.Root)
   ```

Instead of the script, you can apply the patch file: `git apply heart-docker-docker-v29.9.3.patch` (in `results/*/heart-docker.patch`).

## 3. Test and build the built-in version

```sh
go vet ./daemon/heart/ && go test -race -count=1 ./daemon/heart/
CGO_ENABLED=0 go build -trimpath -tags "netgo osusergo static_build exclude_graphdriver_btrfs" \
  -ldflags "-s -w -X github.com/moby/moby/v2/dockerversion.Version=29.9.3-heart -X github.com/moby/moby/v2/dockerversion.GitCommit=$(git rev-parse --short HEAD)-heart" \
  -o dockerd-heart ./cmd/dockerd
./dockerd-heart --version             # Docker version 29.9.3-heart, build 70efc0e-heart
```

On our 8-vCPU build machine this took 45 to 56 seconds, and made a 92 MB static binary.

## 4. Test and build the sidecar (way B)

```sh
cd ../experimentalDocker/heart
go test -race -count=1 ./...
CGO_ENABLED=0 go build -trimpath -ldflags "-s -w" -o heart-docker-sidecar ./cmd/heart-docker-sidecar
```

This gives a 6.7 MB static binary. Its options:

| Option | Default | Meaning |
|---|---|---|
| `-engine` | `unix:///var/run/docker.sock` | The Docker socket it reads |
| `-root` | `/var/lib/docker` | Docker's data root, whose disk it watches |
| `-state` | `/var/lib/heart-docker` | Where it keeps its owner key |
| `-addr` | `127.0.0.1:43278` | Where it listens; loopback only, and it refuses any other address |
| `-poll` | `10s` | How often it checks the Engine |
| `-disk-percent` | `90` | The threshold for "disk nearly full" |

## 5. Get Docker's own release of the same version

The tests need containerd, runc and the docker CLI, and way B and the comparison need Docker's stock `dockerd`. All come from Docker's official static release:

```sh
mkdir -p bin/docker
curl -fsSL https://download.docker.com/linux/static/stable/x86_64/docker-29.9.3.tgz | tar xz -C bin/docker --strip-components=1
cp moby/dockerd-heart experimentalDocker/heart/heart-docker-sidecar bin/
```

## 6. Python for the checks

```sh
python3 -m venv venv && venv/bin/pip install "jsonschema[format]>=4.18" referencing
```

## 7. Run the tests

Run as root, because `dockerd` needs it. With `HEART_TEST_BASE` set, every daemon file stays inside that folder.

```sh
sudo HEART_TEST_BASE=$PWD/run PYTHON=venv/bin/python experimentalDocker/run_both.sh $PWD/bin experimentalDocker/results mymachine 3
python3 experimentalDocker/summarize.py          # writes results/SUMMARY.md and SUMMARY.json
```

`run_both.sh` runs each way 3 times. Each run writes the following to `results/final-<label>-<mode>-run<N>/`:

| File | What is in it |
|---|---|
| `results.json` | Every check and measure |
| `results.md` | The same, as a table |
| `e2e.txt` | The console output |
| `heart-rating.json` | The HEART-Rating report |
| Logs | The daemon and sidecar logs |

To run one mode by hand:

```sh
sudo venv/bin/python experimentalDocker/test_heart_docker.py --mode builtin --heart bin/dockerd-heart --stock bin/docker/dockerd --bin $PWD/bin/docker \
     --schemas spec/schemas/1.0 --rating heart-rating/client/heart_rating.py --metrics heart-rating/metrics/heart-metrics.json --out results/one
sudo venv/bin/python experimentalDocker/test_heart_docker.py --mode sidecar --sidecar bin/heart-docker-sidecar --stock bin/docker/dockerd --bin $PWD/bin/docker \
     --schemas spec/schemas/1.0 --rating heart-rating/client/heart_rating.py --metrics heart-rating/metrics/heart-metrics.json --out results/two
```

### What the tests do

**Both ways (21 checks):**

1. **The daemon:** it starts and answers its Engine API. The storage driver is overlay2 where possible, vfs otherwise.
2. **The HEART's replies:**
   - the HEART answers; the Card names `com.docker.engine`;
   - the public Pulse shows only the status, and the owner key adds Docker's own figures;
   - every reply matches the published HEART schemas;
   - unknown paths get 501 or 404, and a wrong key gets 401;
   - the Engine API itself serves no HEART paths;
   - the HEART-Rating client gives its score.
3. **Real faults in real containers** (BusyBox 1.37). Each fault is checked for the right fault code, how fast it was noticed, and that the public view names no container while the owner's view does:

   | Fault | The container | Fault code |
   |---|---|---|
   | Restart loop | `--restart always busybox sh -c 'sleep 1; exit 3'` | `system.main-function-degraded` |
   | Out of memory | `-m 16m --memory-swap 16m busybox tail /dev/zero` | `performance.memory-exhausted` |
   | Failing health check | `--health-cmd false --health-interval 1s --health-retries 1` | `system.main-function-degraded` |

4. **Faults end:** while the faults are open the Pulse says degraded. When the containers are removed, every container alert resolves.
5. **Disk:** with the threshold lowered to 1%, `storage.nearly-full` is raised.

**Built in only:**
- `HEART=off` closes the port, and the Engine works as usual.
- Docker's stock `dockerd` is started for comparison of memory and start-up time.

**Sidecar only (2 more checks, 23 in all):**
- Docker stops: the sidecar reports `system.main-function-lost`, and the Pulse says failing. Docker restarts: the alert resolves.
- The sidecar follows Docker's events again after the restart (a new restart loop is noticed).
- The sidecar stops: its port closes, and Docker and its running containers carry on.

## 8. Our results

These are the 12 final runs: both ways, 3 runs each, on two machines, with the same two binaries.

| Binary | SHA-256 |
|---|---|
| `dockerd-heart` | `c3c61ec35a08d07496e86ae854d572f48b16bb1da6104e756a3bf8d9650629ce` |
| `heart-docker-sidecar` | `dfdb268731fa16dc7a4fa4ef00d6681896679079f6c6a319545e63e44912e4ec` |

Both were built with Go 1.26.10 on the pod, run `20261112-193556`. See `results/SUMMARY.md` for every figure.

| | VM, built in | VM, sidecar | Server, built in | Server, sidecar |
|---|---|---|---|---|
| Checks passed, 3 runs | 21/21 each | 23/23 each | 21/21 each | 23/23 each |
| Restart loop noticed | 2.9 s | 2.9 s | 3.0–3.3 s | 3.0–3.3 s |
| Out-of-memory kill noticed | 0.4 s | 0.4 s | 0.5–0.6 s | 0.5 s |
| Failing health check noticed | 1.4 s | 1.4 s | 1.5 s | 1.5–1.6 s |
| Docker stopped: noticed | not possible | 6.0 s | not possible | 5.5–6.0 s |
| Pulse, median | 0.49 ms | 0.50 ms | 0.99 ms | 0.99 ms |
| Memory cost of the HEART | +10.9 MB (84.9 against 74.0 with `HEART=off`) | 11.1 MB (its own process) | +12.5 MB (87.8 against 75.3) | 10.5 MB |
| HEART-Rating | 8.9 Good | 8.9 Good | 8.9 Good | 8.9 Good |

The two machines:
- **VM:** a cloud virtual machine with 2 vCPU and 7 GB, running Ubuntu 22.04.5 and Linux 6.8.
- **Server:** a 16-core machine with 125 GB, running other applications (load average about 52), with Ubuntu 24.04.3 and Linux 6.8.

The HEART-Rating fails only `P4-CACHE` (no HTTP caching headers), as every HEART template does.

## 9. Try it by hand

**Way A: Docker with the HEART built in.** Use a separate data root so that your own Docker is untouched:

```sh
sudo env PATH=$PWD/bin/docker:$PATH bin/dockerd-heart --data-root /tmp/hd/root --exec-root /tmp/hd/exec \
     -H unix:///tmp/hd/docker.sock --pidfile /tmp/hd/docker.pid --bridge=none --iptables=false --ip6tables=false &
curl -s 127.0.0.1:43278/.well-known/heart-card.json
curl -s 127.0.0.1:43278/heart/v1/pulse
KEY=$(sudo cat /tmp/hd/root/heart/owner.key)          # also printed once in the daemon's log
D="sudo bin/docker/docker -H unix:///tmp/hd/docker.sock"
$D run -d --name crashy --network none --restart always busybox sh -c 'sleep 1; exit 3'
sleep 5; curl -s 127.0.0.1:43278/heart/v1/arrhythmias                       # codes only
curl -s -H "Authorization: Bearer $KEY" 127.0.0.1:43278/heart/v1/arrhythmias # with the container's name
$D rm -f crashy; sudo kill $(sudo cat /tmp/hd/docker.pid)
```

**Way B: a sidecar on any Docker host.** It only reads the socket, and changes nothing in Docker:

```sh
sudo bin/heart-docker-sidecar -engine unix:///var/run/docker.sock -root /var/lib/docker -state /var/lib/heart-docker
```

As a systemd service (`/etc/systemd/system/heart-docker.service`):

```ini
[Unit]
Description=HEART for Docker (sidecar)
After=docker.service
[Service]
ExecStart=/usr/local/bin/heart-docker-sidecar
Restart=always
[Install]
WantedBy=multi-user.target
```

**Privacy:**
- **Without the key:** anyone sees the Card, the Pulse status and the fault codes.
- **With the owner key:** container names, counts and disk figures are added.
- **Never read:** environment variables, logs and files.

## 10. Building on RunPod (optional)

This is how we built. Any Linux machine with Go works just as well.

```sh
set -a; . ~/.config/heart/b2_etc.env; set +a      # RUNPOD_API_KEY and the B2 settings
python3 experimentalDocker/runpod_run.py --minutes 40
```

What it does:
- **The pod:** it starts a CPU pod (8 vCPU, image `golang:1.26.10-bookworm`, about $0.28 an hour) that runs `pod_main.sh`: clone, patch, unit tests, both builds, the tests that work without containers, and the upload of everything to B2.
- **Ending:** it watches B2 for the pod's DONE or FAILED marker, and always terminates the pod at the end.
- **Cost:** a run takes 4 to 6 minutes, about $0.03.

On a pod, containers cannot run, so 12 of 13 checks pass there. The container tests need a real machine (steps 5 to 7).

## 11. Troubleshooting, and what we learnt

| Symptom | Cause | What to do |
|---|---|---|
| `unshare: operation not permitted` | You are inside an unprivileged container (RunPod, most CI runners) | Use a VM or a real machine |
| The out-of-memory kill is not noticed by an older build | On cgroup v2, Docker itself sometimes misses the kernel's out-of-memory kill: `OOMKilled: false`, exit code 137, and no `oom` event. It did in 2 of 3 of our early runs. | This HEART also counts a container killed with SIGKILL at its memory limit, when nobody asked Docker to kill it, as out of memory |
| `storage.nearly-full` stays open | The disk under your data root really is above 90% (ours was, on the VM) | That alert is true. Raise `-disk-percent` or `HEART_DISK_PERCENT` to test without it |
| The HEART does not start, "no unix or plain tcp Engine API listener" | An early build read the listeners without their `unix://` prefix | Fixed: it reads the configured `-H` hosts |
| Port 43278 in use | Another HEART runs there | Set `HEART_ADDR=127.0.0.1:<port>` (loopback only) |

## 12. Where everything is

**In this repository (`experimentalDocker/`):**

| File | What it is |
|---|---|
| `heart/` | The HEART package, its tests, and `cmd/heart-docker-sidecar` |
| `apply_patch.py` | Adds the package and the hook to Docker's source |
| `pod_main.sh`, `runpod_run.py` | The build on a RunPod pod |
| `test_heart_docker.py` | One run of the tests |
| `run_both.sh` | Both ways, 3 runs each |
| `summarize.py` | Combines the runs into `results/SUMMARY.md` and `SUMMARY.json` |
| `results/` | Every run, raw |

**On B2 (`experimentalDocker/`):**

| Folder | What is in it |
|---|---|
| `source/` | The full modified Docker source |
| `bin/docker-v29.9.3/` | Both binaries, with their SHA-256 |
| `patch/` | The patch |
| `scripts/` | These files |
| `results/` | Every run |