HEART Embedded Systems logoHEART

Guide 10: A HEART-Embedded online shop server#

Reading tracks3 Advanced
Advanced (Stop 3A): self-healing Playbooks, A4. Which track am I?

A case study. A small company runs an online shop: a web tier of worker processes, a cache, a database, a disk full of logs, and two outside services it cannot do without, a payment provider and a shipping API. It wants the shop to be a HEART-embedded system: it should know when it is unwell, say so in one standard language, tell the right person, never leak anything it should not, and fix the few problems everyone already knows how to fix, by itself, within limits the owner set. This guide goes step by step from day 1 to a shop that shows Autonomy Level A4, "fixes known problems", for three known problems, and that an owner registers and watches on HEART Hub.

This is HEART for software rather than hardware. There is no sensor, motor or battery: the "body" is a set of processes and the services they depend on.

A working demo comes with it: onlineShopSimulator/, one Python file that simulates customers, a shop server and its HEART, and reports to https://heartcortex.ai.

Status. A design case study with a simulator. No real shop, customer or payment was involved. The simulator demonstrates A4 behaviour; its Card honestly claims only A1, because it is not a rated HEART Full implementation (Step 11).


Before you start: what HEART will and will not do#

A shop's job, taking orders and money, belongs to the shop's own code, reviewed and tested like any other. HEART does not take part in it. It looks after the machinery around it.

Stays in the shop and its own admin toolsGoes to HEART
Taking orders, payments, refundsKnowing whether the shop is serving customers
Deleting or changing orders and customer recordsWatching workers, queue, cache, memory, disk, latency and outside services
Deploying a new version, rolling one backRaising alerts with standard fault codes, and telling a person
Choosing or changing the payment providerThree harmless fixes, by Playbooks the owner adopted: restart a stuck worker, clear a bloated cache, rotate full logs
Adding servers (it costs money)Answering "how is the shop?" in one standard format
The customer databaseNothing: HEART never reads it

The rule behind this split is the standard's approval gate (Part 6, Section 2.3): a write that can be undone may run automatically if the owner allowlisted it; a destructive action always needs a person, and no Playbook can skip that. This HEART goes one step further: it offers no destructive action at all. Deleting orders, refunding, deploying and changing the payment provider are refused with 403 capability-not-allowed for every caller, even the owner, because a person who wants them has the shop's own tools.


Step 1: Decide where HEART runs#

customers ──► load balancer ──► web workers ──► cache, database, disk
                                     │                 payment provider, shipping API
                           HEART sidecar (same host)
                                     └──(outbound)──► HEART Hub, on-call route
PartWhat it doesHEART's place
Web tier (workers behind a queue)Serves pages and checkoutsWatched; the HEART may restart one worker
CacheKeeps pages and query resultsWatched; the HEART may clear it
DatabaseOrders, products, customersWatched from outside (query time). Never read
DiskLogs, sessions, uploadsWatched; the HEART may rotate and compress logs
Payment provider, shipping APIOutside servicesHealth-checked. Never changed
HEART sidecarIts own process on the same host (class Core-M)Runs the HEART

The HEART is a separate process, so a shop that crashes does not take its HEART with it, and a HEART that crashes leaves the shop serving (Part 2, Rule 4: HEART stays out of the critical path). It never sits between a customer and a page.


Step 2: Give the system an identity#

The sidecar serves a Card at /.well-known/heart-card.json:

{ "system": { "id": "com.example.shop.web", "name": "Example shop web tier", "version": "4.2.0" },
  "level": "full", "bindings": ["http-server"], "pulse": { "intervalSec": 30 },
  "modules": ["properties", "control"],
  "shares": { "pulse": "public", "arrhythmia": "public", "property": "private", "action": "internal",
              "rule": "internal", "control": "internal" },
  "autonomy": { "level": 1, "higher": [ { "level": 4, "areas": ["software", "storage", "performance"] } ] },
  "template": "heart/embeddedstandard/python/v1.2611.2" }

That is the Card of a real, rated shop (Step 11). The simulator's Card says level: basic and autonomy.level: 1, because it is not one. The system ID names the product, never a customer. The device ID used by HEART Hub (Step 9) is random.


Step 3: List what the system knows, and how private each item is#

Decide every value's privacy level before writing code.

ValueSourcePrivacyWhy
Healthy or not; open fault codesDerived by HEARTpublicThe only things HEART Hub ever receives (Step 9)
Storefront: open, degraded or closedDerivedpublicWhat a customer could see anyway
Requests per second, error rate, p95 latency, queue backlogLoad balancer, web tierinternalOperations data, for the owner and on-call
Worker health, longest stuck timeWeb tierinternal
Cache hit rate, memory used, disk free, database latencyHost, cache, database timingsinternal
Payment provider and shipping API reachableHealth checksinternalSays who the shop's suppliers are
Orders todayShop (a count)internalBusiness data, but only a number
Revenue todayShop (a sum)privateCommercially sensitive
Actions, Playbooks, the allowlistHEARTinternal (who adopted: private)How the shop is run
Customer names, addresses, emails, baskets, card dataShop databasenot in HEART at allThe HEART does not need them, so it has no way to read them. Card data never leaves the payment provider

Note what is not an alert: a quiet night or a big sale is business, not a fault. Only a problem of the system (workers, queue, resources, outside services) becomes an alert.


Step 4: Say what "doing its main job" means#

The main job of a shop server is: customers who arrive are served. The HEART checks it from the web tier's own numbers:

def check_main_function():
    """CHANGE: return (True, "") if the shop serves customers, else (False, "what is wrong, in plain words")."""
    m = metrics.latest()
    if m.healthy_workers == 0 or (m.requests_per_sec > 1 and m.served_per_sec < 0.1):
        return False, "The shop is not serving any customer."
    return True, ""

A condition must hold for a few seconds before it is raised, because one odd reading is not a fault.


Step 5: The faults it recognises#

Every alert carries a fault code from Part 3, Section 5.4, so the sidecar, the on-call tools and any monitor site read it the same way.

What happensFault codeSeverityPlaybook?
No customer is being servedsystem.main-function-lostcriticalNo: escalated
A worker has not answered for 30 ssoftware.crashedwarningYes: restart-stuck-worker
A worker hangs again after two restarts in 10 minutessoftware.crash-looperrorNo: escalated (it is what a failed Playbook leaves behind)
Memory 90 % or moreperformance.memory-exhaustederrorYes: clear-bloated-cache, if the cache hit rate is below 50 %
Disk 10 % free or lessstorage.nearly-fullwarningYes: rotate-full-logs
Disk 2 % free or lessstorage.fullcriticalYes: rotate-full-logs
Payment provider or shipping API unreachableconnectivity.dependency-unreachableerrorNo: escalated
More requests arriving than can be servedperformance.overloadedwarningNo: escalated (more servers cost money)
More than 500 requests waitingperformance.backlogwarningNo: escalated
p95 handling time over 1.5 s (a slow database)performance.slowwarningNo: escalated
Sidecar silent for three Pulse intervalssystem.flatlineRaised by HEART Hub

The thresholds are illustrations. A real shop takes them from its own service levels.


Step 6: Teach it three known fixes (Playbooks and the allowlist)#

An A4 system "fixes problems a person taught it about, then reports" (Part 2, A4). Two things make that safe, and both are set by a person:

  1. Playbooks: rules the owner adopted, "when this happens, do that". A Playbook runs only if it is enabled and its status is adopted; it may only ever run the one capability it names (Part 6, Section 4.2).
  2. The allowlist: the owner's list of actions that may run without asking, for initiator rule, each with an envelope: the range of every argument, maxPerDay (most automatic runs in 24 hours) and cooldownSec (least time between two).
PlaybookWhenThenEnvelope
restart-stuck-workerA worker stuck for 30 s (software.crashed)restartWorker {worker}worker 0 to 7; 6 a day; 60 s apart
clear-bloated-cacheMemory 90 % or more (performance.memory-exhausted), and the capability's interlock: hit rate below 50 %clearCache {percent: 100}1 to 100 %; 2 a day; 120 s apart
rotate-full-logsDisk 10 % free or less (storage.nearly-full, storage.full)rotateLogs {keepDays: 7}3 to 30 days; 4 a day; 300 s apart

Why these three. Each is what an on-call engineer does first, half asleep, without thinking: it is reversible, cheap, and does not touch money or data. Each is write, not destructive, and not physical (a server has nothing physical to move). That is exactly what the allowlist is for.

The interlock. Clearing a cache that is working (high hit rate) would only slow the shop and load the database. So clearCache carries an interlock, cache_hit_rate < 50, checked immediately before it runs, for every caller. When it fails the answer is 409 guard-failed, even for the owner.

When the fix does not help. After each run the HEART watches for a while (judgeSec). If the problem comes back, the fix did not help. Twice in a row, and the Playbook is suspended: from then on it escalates instead of looping. A worker that hangs again after every restart is not stuck, it is broken (a poison job, a bad release), and restarting it forever would hide that. Only a person can adopt the Playbook again (PUT /heart/v1/tending/rules/{id}, with proof of presence).

Every run is recorded. Each run, and each refusal, is an Action (initiator: rule, risk: write, physical: false, succeeded, failed or denied with its reason), linked to its alert, with the Playbook that allowed it and the result (Part 2, Rule 5). The owner reads them at GET /heart/v1/actions, and the Playbooks at GET /heart/v1/tending/rules; both are read only through the API.

Where Playbooks come from. A vendor may ship these three, but they start proposed. The shop owner adopts each one, in person, and adds its allowlist entry. The HEART never adopts its own Playbooks and never widens its own allowlist (Part 2, Rule 2). In the simulator, the owner's adoption at setup is simulated.


Step 7: Commands, approvals and Stop#

RequestWhoWhat happens
A Playbook's action, from its PlaybookruleRuns if on the allowlist for rule and inside its envelope and limits; else refused (denied) and escalated
A Playbook's action, by hand, with the owner keyhumanNot physical, so it runs at once, no approval (POST /heart/v1/capabilities/{name}/invoke). Interlocks still apply
Delete orders, refund, deploy, change the payment provideranyone403 capability-not-allowed, always
Any other change (PUT, PATCH, DELETE)anyone403 capability-not-allowed
Stop (POST /heart/v1/tending/stop)the ownerAt once, no approval, no internet: max 2, so mode observe. Playbooks are refused with autonomy-stopped; noticing and escalating go on
Turn automatic action back on (PUT /heart/v1/tending/autonomy, {"max": 4})a person, presentRaising the ceiling max is destructive: a key alone gets 403 presence-required. The simulator's proof of presence is a local button (touch button.press beside it); a real shop uses a passkey (WebAuthn)

Every change carries an Idempotency-Key, so a retried request never restarts a worker twice.


Step 8: Tell a person#

A4 must tell a person afterwards what it noticed, what it did and whether it worked; and for anything no Playbook covers, it falls back to telling a person first. Change escalate() to the shop's on-call route:

  1. On call: a page, a chat message or an email, with the alert's explanation: "The payment provider is down: customers can browse but cannot pay."
  2. Afterwards, for a fix: "Worker 3 had not answered for 35 s. Playbook restart-stuck-worker restarted it. It works again."
  3. When a Playbook gives up: "Restarting worker 5 did not help twice in a row. Playbook suspended. A person must look."

An alert a Playbook is handling has status tending; one handed to a person has escalated, with the reason. HEART Hub (Step 9) adds an email when the shop goes silent.


Step 9: Register and watch it on HEART Hub#

  1. First start. The sidecar makes a random device ID, claim code and key, keeps them, and shows the claim code only on its console.
  2. Sign in. The owner opens https://heartcortex.ai/login and enters an email address. A 6-digit code arrives; there are no passwords.
  3. Register. At https://heartcortex.ai/register?id=<device ID> the owner enters the claim code.
  4. Watch. The device's page, https://heartcortex.ai/device/<device ID>, shows the status and the alerts, tending, escalated or resolved, private to the owner until they share it.
  5. Silence. If no report arrives for three Pulse intervals, the page shows a Flatline and the owner gets one email.

Only the public Pulse and public alerts are sent, and the site keeps only the status, timing and fault codes: never traffic, revenue, an Action or a worker number. A shop with many servers, or a hosting company, can run its own monitor site (the code is open, in heart-monitor/).


Step 10: Try it with the simulator#

python3 onlineShopSimulator/onlineShopSimulator.py --scenario all

It prints a device ID and claim code; register them as in Step 9 and watch the page while it stages each fault in turn: a stuck worker (fixed by its Playbook), a disk filling with debug logs (fixed by its Playbook), a worker in a crash loop (the Playbook gives up and escalates), a cache leak (fixed twice, then refused at the daily limit), a payment outage, a flash sale and a slow database (escalated). Details and every option are in onlineShopSimulator/README.md.

Simulated hereA real shop needs
Customers as a request rate with a daily curve and flash salesIts real traffic, read from the load balancer
Workers, queue, cache, database, disk, outside services as numbersReaders for its real metrics and health checks
restart_worker, clear_cache, rotate_logs change numbersThe same three, against its real tools (process manager, cache, logrotate)
Playbooks adopted at setupAdoption by the owner, in person, through PUT /tending/rules/{id}
A local file as the buttonA passkey (WebAuthn) or another presence method of API Standard 6.3
The console as the escalation routeThe on-call route
A Card claiming Basic, A1HEART Full, rated, before it claims A4

Step 11: Test it, rate it and claim only what passed#

  1. Run the HEART-Rating client (heart-rating/client/heart_rating.py) against the sidecar, and the hands-on tests of Part 9: cause each fault in Step 5 on a staging server, check each Playbook fires once, stays inside its envelope, stops at maxPerDay, waits cooldownSec, gives up when its fix does not help, and is refused after Stop.
  2. Check every destructive request is refused, and that turning automatic action back on needs a present person.
  3. Publish a Declaration with every result.
  4. Claim HEART Full, Autonomy Level A1 · A4 for software, storage and performance.
LevelFits a shop server?
A1 (notices and tells a person)Yes, everywhere, including the main job
A2 (says why, with a confidence)Yes, at Connected: "p95 rose from 100 ms to 3.6 s while database time per page rose to 900 ms: slow database, confidence high"
A3 (suggests a fix, a person approves)Yes, at Full: "add two workers for the flash sale?", "fail over to the backup payment provider?" (still never a Playbook: the owner decides each time)
A4 (fixes known problems)Yes, for three problems: this guide. Needs Full, an adopted Playbook in each area, and an allowlist entry for rule for each
A5 (works out new fixes)Possible for low-risk writes only, with the ai and evolving modules and an allowlist for ai; never for money, data or deploys

What the case study shows#

  • HEART works for software too. Workers, caches, disks and outside services have faults with standard codes, exactly like sensors and batteries.
  • A4 is narrow on purpose. Three reversible fixes that any on-call engineer would make, each with a range, a daily limit and a cooldown. Everything else goes to a person.
  • A Playbook must know when to stop. A fix that does not help is suspended, so the HEART escalates a crash loop instead of hiding it with endless restarts.
  • Some things are never HEART's. Orders, refunds, deploys and the payment provider are refused outright, and customer data is not even within its reach.
  • Claim what passed. The simulator shows A4 behaviour but claims A1; a real shop claims A4 only for the areas it was rated in.