# Online shop server: on a real system **A web shop on a Linux server with a HEART beside it: the server we assume, where everything runs, the code and configuration, how to install it, what to check before going live, and what the shop owner sees.** > **For illustration only.** The server, the shop and its metrics below are assumptions, chosen so the steps can be concrete. Your shop will differ. This system has not been built: no real shop, customer or payment was involved. The Python template and `heartd` are tested on computers with simulated devices only, and have not run beside a real shop. Check every step on your own staging server. | | | |---|---| | **Placement** | A: on the shop's own server, as its own process beside the shop (a sidecar). Nothing here is safety-certified. | | **Start from** | The [Python template](/heart-templates/README.md) (`heart/embeddedstandard/python/v1.2611.3`), with the fault rules of the simulator. `heartd` for the Playbooks (Section 3.6). | | **Claim** | HEART Basic, Autonomy Level A1. The three Playbooks (A4) need HEART Full and a rating first (Section 3.6). | | **Try it first** | [The simulator](/onlineShopSimulator/README.md) and the [design case study](/guides/Online-Shop-Server.md) | --- ## 1. The system we assume | Part | Assumed choice | Notes | |---|---|---| | Server | One Linux virtual machine, Ubuntu 24.04 LTS, 4 vCPU, 8 GB RAM | Any Linux with systemd and Python 3.8 or newer will do | | Shop | A Python web shop (Django or Flask), run as 8 worker processes: systemd units `shop-worker@0` to `shop-worker@7`, grouped by `shop.target` (each worker has `PartOf=shop.target`) | The simulator also has 8 workers | | Front | nginx on ports 80 and 443, in front of the workers | The only thing open to the internet | | Cache | Redis, with the page cache in its own database | | | Database | PostgreSQL, on another server | The HEART never connects to it. It sees only query time, from the shop's metrics | | Logs | `/var/log/shop`, rotated by logrotate | On the same disk as the shop | | Outside services | A payment provider and a shipping API, over HTTPS | The shop checks them itself | | Metrics | The shop serves a small JSON document on loopback: `http://127.0.0.1:8081/internal/heart-metrics` | **You write this.** It is not part of any template. Totals only, never a customer | | Drain | Two loopback endpoints in the shop: `POST /internal/drain` (take no new orders) and `POST /internal/undrain` | **You write these too.** The drain flag lives in memory, so a restarted shop takes orders | | On-call | A chat or pager webhook | Where `escalate()` sends alerts | | Network | The HEART API on `127.0.0.1:43278` only; outbound HTTPS to the HEART Hub site and the webhook | Port 43278 is never opened in the firewall | ### Example metrics document ```json {"workers": ["ok", "ok", "ok", "stuck", "ok", "ok", "ok", "ok"], "stuckSec": [0, 0, 0, 41, 0, 0, 0, 0], "requestsPerSec": 31.0, "servedPerSec": 30.8, "errorRatio": 0.002, "p95Ms": 103, "queue": 0, "memoryUsedPct": 60, "cacheHitPct": 93, "paymentOk": true, "shippingOk": true, "runningOrders": 2} ``` The names are ours. `runningOrders` is the number of checkouts in progress, which a graceful restart waits for. There is no customer, basket, order number or revenue in it, so none can reach the HEART. --- ## 2. Where everything runs ```text ┌──────────────────────────── Linux server (VM) ─────────────────────────────┐ │ nginx :443 ──► shop-worker@0..7 (shop.target) ──► Redis; PostgreSQL elsewhere│ │ │ loopback: /internal/heart-metrics, /internal/drain │ │ ▼ read only │ │ heart.service (user heart): checks every 30 s, API on 127.0.0.1:43278 ───────────► HEART Hub │ └─ may restart shop.target, only after a graceful drain ───────► on-call webhook └──────────────────────────────────────────────────────────────────────────────┘ (outbound only) /var/lib/heart (mode 0700): heart-monitor.json (device ID, claim code, key) · heart-lifecycle.json (owner key, Actions) ``` - **A separate process.** A shop that crashes does not take its HEART with it, and a HEART that crashes leaves the shop serving (Part 2, Rule 4). It never sits between a customer and a page. - **No customer data.** The HEART reads only the metrics document and the free space on the disk. It has no database login, and card data never leaves the payment provider. - **One action, through the API:** a graceful restart of the shop, with the owner key. Orders, refunds, deploys and the payment provider stay in the shop's own admin tools. The three automatic fixes come later, with Playbooks (Section 3.6). --- ## 3. The code Copy `heart.py` from the Python template to `/opt/heart/heart.py`, and change it as below. Section 3.5 adds the other fault codes of the case study. | What | Where in `heart.py` | |---|---| | Identity | The four lines marked `CHANGE` (3.1) | | Main job | `check_main_function()` (3.2) | | Escalation | `escalate()` (3.3) | | Drain and restart | `before_lifecycle()`, `carry_out()` and `LIFECYCLE_OPS` (3.4) | | The other fault codes | A new `check_rules()`, called from `tend()` (3.5) | | HEART Hub, lifecycle | Environment variables in the systemd unit (Section 4): `HEART_MONITOR_URL`, `HEART_LIFECYCLE` | Add one import at the top: `import subprocess`. ### 3.1 Identity ```python SYSTEM_ID = "com.example.shop.web" # CHANGE: as the case study's Card NAME = "Example shop web tier" # CHANGE VERSION = "4.2.0" # CHANGE: the shop's version INSTANCE = "web-1" # CHANGE: must not identify a person ``` Keep `TEMPLATE` as it is. It records where the code started. ### 3.2 The main job The case study's main job is: **customers who arrive are served.** A payment outage is not a lost main job: customers can still browse, and it has its own fault code (3.5). ```python METRICS_URL = "http://127.0.0.1:8081/internal/heart-metrics" # assumed: your shop serves it on loopback def read_metrics(): with urllib.request.urlopen(METRICS_URL, timeout=5) as r: return json.load(r) def check_main_function(): """CHANGE: return (True, "") while the shop serves customers, else (False, "what is wrong, in plain words").""" try: m = read_metrics() except Exception: # noqa: BLE001 return False, "The shop does not answer its own health check." if m["workers"].count("ok") == 0 or (m["requestsPerSec"] > 1 and m["servedPerSec"] < 0.1): return False, "The shop is not serving any customer." return True, "" ``` **Do not check the main job with a test order.** A synthetic checkout would create orders and payments. Read what the shop already measures. ### 3.3 Escalation `tend()` calls `escalate()` while it holds the template's lock, so send in the background: ```python ONCALL_URL = os.environ.get("HEART_ONCALL_WEBHOOK") # our own variable, not a template option def escalate(alert): """CHANGE: tell the on-call person, in plain words.""" text = f"Shop: {alert['summary']}. {alert.get('explanation', '')}" print(f"HEART escalate: {text}", file=sys.stderr) # also in the journal if ONCALL_URL: req = urllib.request.Request(ONCALL_URL, data=json.dumps({"text": text}).encode(), headers={"Content-Type": "application/json"}) threading.Thread(target=lambda: urllib.request.urlopen(req, timeout=10), daemon=True).start() ``` The message holds the fault's summary and explanation, nothing about customers. Check what your chat or pager service keeps. ### 3.4 Drain and restart The template restarts its own program. For a shop, a restart must first **drain** (take no new orders, let the running ones finish), then restart the shop, as the simulator does (API Standard 13.2 rule 3). If the running orders do not finish by `timeoutSec`, the restart fails and is **never forced**. ```python LIFECYCLE_OPS = ("restart",) # changed from ("restart", "shutdown"): a shop is not shut down through HEART SHOP_ADMIN = "http://127.0.0.1:8081/internal" # assumed: your shop's loopback admin endpoints def shop_post(path): urllib.request.urlopen(urllib.request.Request(SHOP_ADMIN + path, data=b"{}", method="POST"), timeout=5) def before_lifecycle(operation, deadline): """CHANGE: drain. New checkouts get 503 'please try again in a minute'; running ones finish.""" shop_post("/drain") try: while time.time() <= deadline: if read_metrics()["runningOrders"] == 0: return True time.sleep(0.5) except Exception as e: # noqa: BLE001 print(f"HEART: drain check failed: {e}", file=sys.stderr) shop_post("/undrain") # not ready in time: take orders again. The restart fails; it is not forced. return False def carry_out(operation): """CHANGE: restart the shop, then this program, as the template does (its Action is then marked done).""" sys.stderr.flush() if operation == "restart": subprocess.run(["/usr/bin/systemctl", "restart", "shop.target"], timeout=120) os.execv(sys.executable, [sys.executable] + sys.argv) os._exit(0) ``` `systemctl restart shop.target` restarts every worker with `PartOf=shop.target`. The `heart` user may do this, and nothing else, through one polkit rule (Section 4). A **force restart** (`"mode": "force"`) is destructive. Leave `HEART_MONITOR_APPROVALS` off, and the template refuses it with `403 presence-required`: a person who needs one runs `systemctl restart shop.target` at the server. Section 9 shows the other choice. ### 3.5 The other fault codes The template raises only `system.main-function-lost`. Copy the case study's other rules from `RULES` in `onlineShopSimulator.py`, with your own thresholds: ```python SHOP_RULES = [ # CHANGE: code, severity, condition, summary. The thresholds are the case study's illustrations. ("software.crashed", "warning", lambda m: max(m["stuckSec"]) >= 30, "A web worker has stopped answering"), ("performance.memory-exhausted", "error", lambda m: m["memoryUsedPct"] >= 90, "Memory is nearly used up"), ("storage.nearly-full", "warning", lambda m: 2 < m["diskFreePct"] <= 10, "The disk is nearly full"), ("storage.full", "critical", lambda m: m["diskFreePct"] <= 2, "The disk is full: logs and uploads fail"), ("connectivity.dependency-unreachable", "error", lambda m: not (m["paymentOk"] and m["shippingOk"]), "An outside service the shop needs cannot be reached"), ("performance.backlog", "warning", lambda m: m["queue"] > 500, "Requests are piling up in the queue"), ("performance.slow", "warning", lambda m: m["p95Ms"] > 1500, "Pages take much longer than normal"), ] def check_rules(): """Ours, not the template's. Called from tend() inside its lock. Returns True if any rule holds.""" try: m = read_metrics() except Exception: # noqa: BLE001 the main-job check reports a shop that does not answer return False st = os.statvfs("/var/log/shop") m["diskFreePct"] = 100 * st.f_bavail / st.f_blocks any_open = False for code, severity, holds, summary in SHOP_RULES: active = holds(m) open_alert = next((a for a in state["alerts"] if a["code"] == code and a["status"] != "resolved"), None) if active and open_alert is None: alert = {"id": str(uuid.uuid4()), "kind": f"{SYSTEM_ID}.{code.split('.')[1]}", "code": code, "severity": severity, "status": "escalated", "detectedAt": now(), "summary": summary, "explanation": ""} state["alerts"] = (state["alerts"] + [alert])[-100:] escalate(alert) if not active and open_alert is not None: open_alert.update(status="resolved", resolvedAt=now()) any_open = any_open or active return any_open ``` In `tend()`, change the line that sets the status to: ```python state["status"] = "failing" if not ok else ("degraded" if check_rules() else "healthy") ``` For the plain-words explanations, copy `EXPLAIN` from the simulator as well. `performance.overloaded` needs the shop's capacity, so add it when your metrics give one. `software.crash-loop` belongs with the Playbooks (3.6). The template checks once every 30 seconds, so a condition is raised on the first check that sees it. The case study holds a condition for a few seconds first; to do the same, raise only on two checks in a row. ### 3.6 The three Playbooks (A4): HEART Full, with `heartd` The Python template has **no Playbooks, allowlist or Stop**. They are HEART Full. The simulator's `Heart` class has them, but nobody has rated it. The reference implementation that has them is [`heartd`](/heart-core-rs/README.md), configured as in its [configuration guide](/heart-core-rs/docs/CONFIGURATION.md). This is a sketch for `rotate-full-logs`; check it with `heartd check` before relying on it. The capability, in `custom_dir` (`custom/rotate-logs.json`). It runs a program from an argument list, never a shell: ```json {"name": "com.example.shop.rotateLogs", "kind": "command", "title": "Rotate and compress the logs", "description": "Start new log files and compress the old ones.", "inputSchema": {"type": "object", "required": ["keepDays"], "additionalProperties": false, "properties": {"keepDays": {"type": "integer", "minimum": 3, "maximum": 30}}}, "risk": "write", "privacy": "internal", "physical": false, "callers": ["human", "rule"], "native": {"protocol": "process", "timeoutMs": 60000, "request": {"argv": ["/usr/bin/systemctl", "start", "shop-logrotate@{{arguments.keepDays}}.service"]}}} ``` The Playbook, in the `rules` file. The vendor ships it **`proposed`**: it does nothing until the shop owner changes it to `adopted` (in this file, adoption is by whoever installs it): ```json [{"id": "rotate-full-logs", "enabled": true, "status": "proposed", "description": "When the disk has 10 % or less free, rotate and compress the logs.", "when": {"signal": "diskFreePct", "op": "<=", "value": 10}, "then": {"raise": {"code": "storage.nearly-full", "severity": "warning", "summary": "The disk is nearly full"}, "capability": "com.example.shop.rotateLogs", "arguments": {"keepDays": 7}}}] ``` The allowlist entry and its envelope, in `heartd.toml`. Only the owner adds it, and the HEART never widens it: ```toml [core] mode = "act" [[core.allowlist]] target = "com.example.shop.rotateLogs" initiators = ["rule"] [core.allowlist.envelope] maxPerDay = 4 cooldownSec = 300 [core.allowlist.envelope.arguments.keepDays] min = 3 max = 30 ``` The shop sends `diskFreePct` (and its other numbers) to `heartd` through `host_socket`, one line of JSON each: `{"vital": "diskFreePct", "value": 8.5}`. Register the owner's passkey under `[presence]`, so that turning automatic action back on after a Stop needs a person, as the case study's Step 7 asks. **Open points to check before you rely on this:** - `restart-stuck-worker` needs the number of the stuck worker. A rule's `arguments` are fixed, so one way is one rule per worker, each matching its own `component`. Test it. - `clear-bloated-cache` has an interlock, cache hit rate below 50 %. That needs the hit rate as a signal `heartd` can read immediately before it runs. - `heartd` takes port 43278 in place of `heart.py`, and **does not report to HEART Hub** in this release. - Claim A4 only after HEART-Rating and the case study's Step 11 tests pass: **HEART Full, Autonomy Level A1 · A4 for software, storage and performance**. --- ## 4. Build and install There is nothing to build. You need Python 3.8 or newer (Ubuntu 24.04 has 3.12). ```sh sudo useradd --system --home-dir /var/lib/heart --shell /usr/sbin/nologin heart sudo install -d -m 0755 /opt/heart /etc/heart sudo install -m 0644 heart.py /opt/heart/heart.py sudo install -m 0600 heart.env /etc/heart/heart.env # HEART_ONCALL_WEBHOOK=https://... (a secret) sudo install -m 0644 heart.service /etc/systemd/system/heart.service sudo install -m 0644 50-heart-shop.rules /etc/polkit-1/rules.d/50-heart-shop.rules echo "net.ipv4.ip_local_reserved_ports = 43278" | sudo tee /etc/sysctl.d/99-heart.conf sudo sysctl --system sudo systemctl daemon-reload sudo systemctl enable --now heart ``` Reserving the port stops Linux lending 43278 to an outgoing connection (from the `heartd` configuration guide). ### The systemd unit (`/etc/systemd/system/heart.service`) ```ini [Unit] Description=HEART for the shop After=network-online.target Wants=network-online.target [Service] ExecStart=/usr/bin/python3 /opt/heart/heart.py User=heart Group=heart StateDirectory=heart StateDirectoryMode=0700 WorkingDirectory=/var/lib/heart Environment=HEART_MONITOR_URL=https://heartcortex.ai Environment=HEART_LIFECYCLE=1 EnvironmentFile=/etc/heart/heart.env Restart=on-failure RestartSec=2 # The HEART must never take resources from the shop. MemoryMax=64M CPUQuota=10% Nice=10 NoNewPrivileges=true ProtectSystem=strict ProtectHome=true PrivateTmp=true PrivateDevices=true [Install] WantedBy=multi-user.target ``` - `HEART_MONITOR_URL` set turns HEART Hub reporting on, as `MONITOR = True` would. Use your own monitor site's address if you run one. - The template keeps `heart-monitor.json` and `heart-lifecycle.json` in its working directory, so they land in `/var/lib/heart`, mode 0600. - The limits follow `packaging/heartd.service` in `heart-core-rs`. 64 MB for Python is our guess: measure it. ### The polkit rule (`/etc/polkit-1/rules.d/50-heart-shop.rules`) The `heart` user may restart the shop, and start the log rotation unit that the `heartd` sketch uses. Nothing else: ```js polkit.addRule(function (action, subject) { if (action.id == "org.freedesktop.systemd1.manage-units" && subject.user == "heart") { var unit = action.lookup("unit"), verb = action.lookup("verb"); if (verb == "restart" && unit == "shop.target") return polkit.Result.YES; if (verb == "start" && /^shop-logrotate@[0-9]+\.service$/.test(unit)) return polkit.Result.YES; } }); ``` ### First start ```sh journalctl -u heart -b --no-pager ``` ```text HEART: owner key (shown once, keep it safe): ... {"heart": "heart/embeddedstandard/python/v1.2611.3", "listening": "127.0.0.1:43278"} HEART Hub: register this device at https://heartcortex.ai/register?id=5c1e9a07d3b2f468 with claim code AB3CD-EF4GH-JK5LM-NP6QR ``` Put the owner key in your secret store. Then check it, on the server: ```sh curl -s http://127.0.0.1:43278/heart/v1/pulse KEY=... # the owner key curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:43278/heart/v1/arrhythmias ``` **No reverse proxy.** Do not route `/heart/` through nginx: the API stays on loopback. An on-call person who needs it from elsewhere uses an SSH tunnel: `ssh -L 43278:127.0.0.1:43278 shop-1`. ### Variant: a sidecar container (Docker Compose) If the shop runs in containers, run the HEART as its own container that shares the shop container's network, so it reaches the shop's loopback endpoints: ```yaml services: web: image: registry.example.com/shop:4.2.0 volumes: ["shop-logs:/var/log/shop"] heart: image: python:3.12-slim command: ["python3", "/opt/heart/heart.py"] working_dir: /var/lib/heart network_mode: "service:web" # the shop's 127.0.0.1 is its 127.0.0.1 environment: HEART_MONITOR_URL: https://heartcortex.ai volumes: - ./heart.py:/opt/heart/heart.py:ro - heart-state:/var/lib/heart - shop-logs:/var/log/shop:ro read_only: true restart: unless-stopped volumes: shop-logs: heart-state: ``` - **Leave `HEART_LIFECYCLE` off here.** The container cannot restart the shop without Docker's socket, and must never be given it. Restarts are your orchestrator's. - When the `web` container is recreated, restart the `heart` container too, because it shares `web`'s network. - The disk it sees is the volume's. To watch the containers themselves, see [Docker with a HEART](/experimentalDocker/README.md) (experimental). --- ## 5. Before going live 1. **One identity per server.** The device ID, claim code and key are made at first start in `/var/lib/heart`. If you make a VM image or snapshot after that, every copy shares them. Delete `heart-monitor.json` and `heart-lifecycle.json` before making an image. 2. **The journal holds the secrets once.** The owner key and the claim code are printed at first start. Read them, store them, and keep the journal readable only by administrators (the `adm` and `systemd-journal` groups on Ubuntu). 3. **Keep `/var/lib/heart` in backups,** or a rebuilt server is a new device that must be registered again. 4. **Firewall:** 43278 and 7685 closed from outside; outbound HTTPS to the monitor site and the webhook open. 5. **Time:** keep the clock synchronised (Ubuntu's `systemd-timesyncd` does). Signed decisions from the monitor site are refused if more than 300 seconds old. 6. **Staging first.** Run every test of Section 7 on a staging copy of the shop, then the [HEART-Rating](/heart-rating/README.md) client, and claim only what passed. --- ## 6. What the owner sees 1. **Registration.** The owner signs in at heartcortex.ai with an emailed code, opens the `register?id=...` address from the journal, and types the claim code. The page shows **Healthy**. It is private until the owner shares it. See the [owner's manual](../HEART-Monitor-Owner-Manual.md). 2. **Every day.** Nothing. The HEART checks the shop every 30 seconds. 3. **When something goes wrong:** | Real event | The HEART notices | The owner and on-call see | |---|---|---| | A release makes every worker fail at start | `system.main-function-lost`: "The shop is not serving any customer." | **Failing** on the page; a message on call | | One worker hangs on a request that never finishes | `software.crashed`, after 30 s stuck | A warning, **Degraded**. With A1, a person restarts the worker | | Debug logging left on fills the disk | `storage.nearly-full` at 10 % free; `storage.full` at 2 % | A warning, then **critical** | | A cache eviction bug uses up memory | `performance.memory-exhausted` at 90 % | An error, on call | | The payment provider is down | `connectivity.dependency-unreachable` | An error: "customers can browse but cannot pay". Nobody fixes it through HEART | | A flash sale | `performance.backlog` above 500 waiting | A warning. Adding servers costs money: the owner decides | | A slow database | `performance.slow`, p95 over 1.5 s | A warning | | A quiet night, or a big sale that is served | Nothing: business, not a fault | Nothing | | The server is down, or the HEART stopped | Nothing can be sent | After 90 seconds the page shows **Not heard from**, and one email | 4. **What the site keeps:** the status, the timing and each alert's fault code, severity and times. Never traffic, revenue, an Action, a worker number or anything about a customer. The summary and explanation reach the on-call route and the owner-key API only. 5. **A planned restart.** The owner, or a deploy script, asks with the owner key: ```sh curl -s -X POST -H "Authorization: Bearer $KEY" -d '{"timeoutSec": 60}' \ http://127.0.0.1:43278/heart/v1/lifecycle/restart # 202, with the Action in Location ``` Customers who try to pay during the drain see the shop's 503 message ("please try again in a minute"), their baskets kept. --- ## 7. Test it on staging Cause every fault for real, on a staging copy with no real customers or payments, and check the right fault code appears within a minute on the API and on the HEART Hub page. | Fault | How to cause it on staging | Expect | |---|---|---| | Shop down | `sudo systemctl stop shop.target` | `system.main-function-lost`, **Failing**, a message on call | | A worker hangs | `sudo kill -STOP ` for a minute, then `kill -CONT` | `software.crashed`, then resolved | | Disk nearly full | `fallocate -l /var/log/shop/fill.bin` until 10 % or less is free; delete it after | `storage.nearly-full`; at 2 %, `storage.full` | | Memory | Use memory with a stress tool, for example `stress-ng --vm 1 --vm-bytes 90%` | `performance.memory-exhausted` | | Payment provider down | Point staging's payment setting at an address that does not answer, or block it in the firewall | `connectivity.dependency-unreachable` | | Backlog | A load test at six times normal traffic | `performance.backlog` | | Slow database | In a staging `psql` session, `BEGIN; LOCK TABLE IN ACCESS EXCLUSIVE MODE;` for 2 minutes | `performance.slow` | | HEART stopped | `sudo systemctl stop heart` | The shop keeps serving; **Not heard from** after 90 s and one email; then **Healthy** after a start | | HEART crashes | `sudo kill -9 ` | systemd starts it again in 2 s, with the same device ID | | Graceful restart | `POST /heart/v1/lifecycle/restart` with `{"timeoutSec": 60}` and no slow order | `202`; the shop restarts; the Action ends `succeeded` | | Restart that cannot finish | Start a staging checkout slower than 10 s, then ask with `{"timeoutSec": 10}` | The Action ends `failed`, "It was not forced"; the shop takes orders again | | Force restart | `{"mode": "force"}` | `403 presence-required` | | Shutdown | `POST /heart/v1/lifecycle/shutdown` | `501 not-implemented` (not offered) | | Any other change | `curl -X PUT http://127.0.0.1:43278/heart/v1/properties` | `501 not-implemented` from the template. Nothing changes | Then run the HEART-Rating client against `http://127.0.0.1:43278` from the server, and claim only what passed: HEART Basic, Autonomy Level A1. --- ## 8. Restarts and deploys: what to expect - **Deploys are not HEART's.** Your deploy tool rolls out a new version. If it stops all workers at once, the HEART sees the shop stop serving and raises `system.main-function-lost` for that moment. Deploy one worker at a time, or have the deploy script ask for a graceful restart through the API. - **A graceful restart can fail, on purpose.** A checkout still running at `timeoutSec` stops it, and the shop goes on taking orders. Ask again, or restart by hand at the server. - **A restart through HEART also restarts HEART.** Its API is away for a few seconds, and the Action reads `succeeded` once it is back. The page does not go to **Not heard from**, which needs 90 seconds of silence. - **With A1, people still do the fixing.** The HEART tells on-call within 30 seconds that a worker hangs or the disk fills. A person restarts the worker or rotates the logs, until the Playbooks of Section 3.6 are adopted and rated. --- ## 9. Going further - **Approve a force restart from anywhere.** Set `HEART_MONITOR_APPROVALS=1`. A force restart then waits as `awaiting-approval`, and the owner approves it on the monitor site with a code emailed to them (presence method `remote-site`). Without it, a force restart is done at the server. See the [owner's manual, Section 9](../HEART-Monitor-Owner-Manual.md#9-approvals-a-device-asks-you-first). - **Explanations from an AI model.** Set `HEART_BRAIN=1` with the monitor on. The monitor site acts as the Command Center and holds the model keys; an answer is shown as a suggestion and never acted on. See [the brain lease](/heart-templates/README.md#an-ai-brain-on-a-lease-brain-module-new-in-v126113). - **Several servers.** One HEART per server, each with its own device ID. A hosting company can run its own monitor site from the open code in `heart-monitor/`. - **A Node.js shop.** The JavaScript template (`heart/embeddedstandard/javascript-node/v1.2611.3`) has `checkMainFunction()`, `beforeLifecycle()`, `carryOut()` and `escalate()`, with `HEART_LIFECYCLE`. It does not report to HEART Hub, and it always refuses a force restart. See [restart and sleep](/heart-templates/README.md#restart-and-sleep-lifecycle-new-in-v126113). - **The Playbooks** (Section 3.6), for the narrow A4 of the case study: three reversible fixes, each with a range, a daily limit and a cooldown, suspended when a fix does not help twice in a row.