Online shop server simulator: a HEART that fixes known problems#

A demonstration only. It simulates customers, an online shop server and its HEART. It is not a shop, takes no real orders and holds no customer data.

HEART for software rather than hardware. One Python file, onlineShopSimulator.py, with only the standard library. Run it, register it on HEART Hub with the code it prints, and watch its health on your own private page while it lives a simulated life: quiet nights and busy afternoons, flash sales, workers that hang, a cache that leaks, logs that fill the disk, a payment provider that goes down and a database that turns slow.

Unlike the pacemaker simulator, this HEART changes things. Its owner adopted three Playbooks ("when this happens, do that") and put each fix on the allowlist for initiator rule, with a daily limit and a cooldown. That is the behaviour of Autonomy Level A4, "fixes known problems", shown here as a demonstration (see section 6 for what the Card claims). Everything no Playbook covers is noticed, explained and escalated to a person, and anything destructive is never possible through HEART at all.

The case study behind it, step by step for a real shop, is guides/Online-Shop-Server.md.

On a real device: Online Shop Server, for real shows the hardware we assume, where the HEART runs, the code to change, how to build and install it, the factory steps and what the owner sees. For illustration only.

1. Download and run#

You need Python 3.8 or newer. Nothing to install.

python3 onlineShopSimulator.py --scenario all

It starts at once and prints three things:

  HEART Hub: register this simulated shop server
    1. Sign in at https://heartcortex.ai/login with your email (a 6-digit code is sent to you).
    2. Open https://heartcortex.ai/register?id=3f9c0a71b2d4e865
    3. Enter the claim code  K7Q2M-9XMBA-3RTVW-HP6DZ
    4. Watch it at https://heartcortex.ai/device/3f9c0a71b2d4e865  (private to you until you share it)

10:26:01 HEART API on http://127.0.0.1:43278/heart/v1/pulse  (owner key for private views: ...)
10:26:16    31 req/s · errors  0.2% · p95   103 ms · queue     0 · workers 8/8 · cache  93% · memory  60% · disk free 35.0% · payment up · HEART healthy

Every 15 seconds it prints one line of what the simulated server measures. When something goes wrong, its HEART says so, and says what it did:

10:27:36 shop: worker 3 hangs (staged: a request that never finishes)
10:28:11 HEART tending  software.crashed: A web worker has stopped answering. Worker 3 has not answered for 35 s.
         Its requests time out. Playbook restart-stuck-worker will act.
10:28:11 HEART fixed    software.crashed: Playbook restart-stuck-worker ran restartWorker({"worker": 3}): {"worker": 3, "was": "stuck", "now": "ok"}
10:28:16 HEART resolved   software.crashed: a web worker has stopped answering
10:37:46 HEART error    connectivity.dependency-unreachable: An outside service the shop needs cannot be reached. The payment
         provider is down: customers can browse but cannot pay. Escalated: No Playbook covers this problem: a person must decide what to do.

Stop it with Ctrl-C. The device ID, claim code and key are kept in online-shop-monitor.json (readable only by you), so the same simulated server comes back next time. Delete that file to make a new device.

2. Register it and watch it on HEART Hub#

1. Sign in. Open https://heartcortex.ai/login and enter your email address. A 6-digit code arrives; type it in. There are no passwords.
2. Register. Open the register?id=... link the simulator printed, and enter the claim code. The claim code is the proof that you hold the server: it is shown only on the simulator's console, and the site holds only its hash.
3. Watch. Your server's page, https://heartcortex.ai/device/<ID>, shows its status (healthy, degraded or failing) and its open and recent alerts, each with the plain-language meaning of its fault code. An alert a Playbook is handling shows as tending; one handed to a person shows as escalated. All your devices are at https://heartcortex.ai/devices.
4. Share, if you want. The page is private until you make a share link or make it public.
5. Silence. Stop the simulator: after three missed Pulse intervals (90 seconds) the page shows a Flatline, and you get one email. The site checks for silent devices every 5 minutes, so the email can take a few minutes more.

Until you register it, the site stores nothing. After registering, it can take up to five minutes for the first report to arrive, or less as soon as something changes.

What the site sees. Only the public Pulse and public alerts, and it keeps only the status and timing and each alert's fault code, severity and times. Never a request rate, a memory figure, an order count, revenue, an Action or which worker hung. Customer data is not even in the HEART.

3. The staged faults#

With --scenario, each fault starts at a set time, lasts a while, and is then fixed, either by a Playbook or "by the owner" (outside HEART, as a developer would). Times are in simulated seconds: at --speed 1 they are real seconds.

Without --scenario, faults still happen now and then on random timers, as in life, but rarely: a worker hang every few hours, a payment outage every 12 to 48 hours, a slow database every 6 to 24 hours, a flash sale every day or so.

4. Options#

5. The local HEART API#

The simulator also serves the HEART API on your own computer, as a real system would:

curl http://127.0.0.1:43278/.well-known/heart-card.json         # the Card: what it is
curl http://127.0.0.1:43278/heart/v1/pulse                       # public health
curl http://127.0.0.1:43278/heart/v1/arrhythmias                 # public alerts: codes only
curl http://127.0.0.1:43278/heart/v1/properties                  # public values: storefront open, degraded or closed
KEY=...   # the owner key it printed
H="Authorization: Bearer $KEY"
curl -H "$H" http://127.0.0.1:43278/heart/v1/properties          # all values
curl -H "$H" http://127.0.0.1:43278/heart/v1/arrhythmias         # alerts with explanations, escalations and their Actions
curl -H "$H" http://127.0.0.1:43278/heart/v1/tending/rules       # the three Playbooks (read only)
curl -H "$H" http://127.0.0.1:43278/heart/v1/tending/autonomy    # max, current, mode and the allowlist with its limits
curl -H "$H" http://127.0.0.1:43278/heart/v1/actions             # every Action: who, why, result or refusal (read only)
curl -H "$H" http://127.0.0.1:43278/heart/v1/rhythm              # history
curl -H "$H" http://127.0.0.1:43278/heart/v1/sales               # private: orders and revenue today, totals only

Run a Playbook's action by hand. The owner key makes you a human initiator; the actions are not physical, so they run at once (no approval). Every change carries an Idempotency-Key:

curl -X POST -H "$H" -H "Idempotency-Key: $(uuidgen)" -d '{"arguments":{"worker":2}}' \
     http://127.0.0.1:43278/heart/v1/capabilities/restartWorker/invoke        # 200: a succeeded Action
curl -X POST -H "$H" -H "Idempotency-Key: $(uuidgen)" -d '{"arguments":{"percent":100}}' \
     http://127.0.0.1:43278/heart/v1/capabilities/clearCache/invoke           # 409 guard-failed while the cache is useful

Never possible, with or without the key:

curl -X POST -H "$H" -d '{"arguments":{}}' http://127.0.0.1:43278/heart/v1/capabilities/deleteOrders/invoke   # 403 capability-not-allowed
#   ... the same for refundOrder, deployVersion and changePaymentProvider, and any PUT, PATCH or DELETE

Stop, and start again. The shop promises autonomy level 4. A person sets the ceiling max; the current level is the lower of the two, and the mode follows from it: 4 is act, 2 or below is observe. Stop sets max to 2. It needs only the key and takes effect at once. Raising max again gives authority back, so it needs a person at the server, not just a key: in this simulator, the local button is a file. PUT /heart/v1/tending/autonomy takes a max of 0 to 2 (the same as Stop) or 4; any other value answers 400 invalid-request. The owner's Pulse shows autonomy with max, current, and limitedBy: stopped while max is below 4.

curl -X POST -H "$H" http://127.0.0.1:43278/heart/v1/tending/stop                            # max 2, mode observe: Playbooks are refused
curl -X PUT -H "$H" -H "Idempotency-Key: $(uuidgen)" -d '{"max":4}' \
     http://127.0.0.1:43278/heart/v1/tending/autonomy                                         # 403 presence-required
touch button.press      # in the simulator's working directory, within 2 minutes; used once
curl -X PUT -H "$H" -H "Idempotency-Key: $(uuidgen)" -d '{"max":4}' \
     http://127.0.0.1:43278/heart/v1/tending/autonomy                                         # 200: Playbooks act again

Drain and restart (lifecycle). The shop can stop taking new orders, finish the running ones, and restart (API Standard 13.2 rule 3). The Card lists module lifecycle with operations drain and restart, and wakeSources: ["local"]: a stopped shop is started again by its service manager or a person at the server, never remotely.

A request it takes answers 202 with the Action, and a Location to follow it. Restart Actions carry nextReachableAt.

The restart, simulated. The HEART API closes for about 5 seconds (connections are refused), as a real restart would. Then the shop comes back with its state reloaded: new workers, an empty queue, a cold cache, uptime from 0. Logs, the disk, the day's totals and the HEART's records are kept, as if read back from disk.

Orders, for the demo. POST /shop/checkout is the shop's own storefront, not HEART: a simulated customer pays. It takes {} or {"seconds": 300} (an order that is slow to finish) and holds no customer data. While the shop drains it answers 503 with a clear message: "The shop is about to restart and takes no new orders for a moment. Your basket is kept: please try again in a minute."

curl -X POST -d '{"seconds":300}' http://127.0.0.1:43278/shop/checkout                       # a slow order
curl -X POST -H "$H" -H "Idempotency-Key: $(uuidgen)" -d '{"timeoutSec":10}' \
     http://127.0.0.1:43278/heart/v1/lifecycle/restart                                        # 202 running; fails after 10 s, not forced
curl -X POST -H "$H" -H "Idempotency-Key: $(uuidgen)" -d '{"mode":"force"}' \
     http://127.0.0.1:43278/heart/v1/lifecycle/restart                                        # 202 awaiting-approval
touch button.press      # a person at the server approves it: the shop restarts, the slow order is cut off
curl -H "$H" http://127.0.0.1:43278/heart/v1/lifecycle                                        # phase, allowed, taking orders

Approve, deny, cancel. POST /heart/v1/actions/{id}/decision with the key and an Idempotency-Key:

Refused at once, and no Action is made: an unknown operation (404 not-found), a standard one this shop does not offer, such as shutdown (501 not-implemented), unknown fields, operation in the body not the path's, delaySec above 0 (there is no schedules module) and mode on a drain (400 invalid-request). One lifecycle operation at a time: another answers 409 conflict.

Every message it serves is a valid HEART message: test_e2e.py checks each one against the official schemas.

6. How the code is built#

Customers  ──►  Shop  ◄──(3 capabilities only)──  Heart  ──►  local HEART API      (127.0.0.1)
 arrivals       workers, queue, cache,             watches,    └►  HEART Hub    (outbound only)
                database, disk, payment, shipping  Playbooks

The gate, in order, for each Playbook whose alert is open: the Playbook must be adopted and enabled; the mode must be act (else autonomy-stopped); its capability must be on the allowlist for rule; every argument inside the entry's envelope; the cooldown must have passed (the rule waits); fewer than maxPerDay automatic runs in 24 hours (else refused); then the capability's own interlocks (else guard-failed). Each run, and each refusal, is an Action record (initiator: rule, risk: write, physical: false) linked to its alert. A fix that the problem comes back after (within judgeSec) did not help; twice in a row suspends the Playbook, which from then on escalates. Only a person can adopt it again (restart the simulator; a real product: PUT /heart/v1/tending/rules/{id} with proof of presence).

What is physical here? Nothing. A web server has no motor, heater or valve: restarting a process, clearing a cache and compressing logs are software changes that can be undone, so they are write, not physical. That is why a Playbook may run them on the allowlist, and why the owner's own run needs no presence.

What is destructive? Deleting orders, refunding, deploying a new version, changing the payment provider. HEART does not offer them at all: there is no function in Shop for HEART to call, and the API refuses them with 403 capability-not-allowed for every caller. They stay in the shop's own admin tools, with their own login and review.

What the Card claims. The Card says level: basic, autonomy.level: 1, and the Pulse says autonomy.current: 1. That is honest for a simulator nobody has rated. A real shop that does what this one does would implement HEART Full (the control module, approvals, Stop), pass HEART-Rating, and claim A1 · A4 for software, storage and performance (case study, Step 11). The Playbooks here show what A4 means; they are not a claim.

7. What code to add to make a real product HEART-embedded#

For a real shop, HEART runs as a sidecar on each server (or one per cluster), beside the web tier, never inside the checkout code path: see Step 1 of the case study. Pick a starting point, then add four things.

Starting points

The four things you add

1. Read the server. Replace Customers and Shop with readers for what you already measure: your load balancer's request and error counts, your web server's worker status, your cache's INFO output, the disk, and a health check on each outside service. Read only, except for the three actions in step 2.

``python class ShopReader: """CHANGE: fill these from your own metrics. Read only.""" def refresh(self): m = your_metrics.latest() # Prometheus, your load balancer, your APM self.rps, self.error_rate, self.p95 = m.requests_per_sec, m.error_ratio, m.latency_p95_ms self.queue, self.workers = m.queue_depth, m.worker_states # e.g. "ok" or "stuck" per worker self.memory, self.hit_rate = m.memory_used_pct, m.cache_hit_pct self.disk_free = m.disk_free_pct self.deps = {"payment": m.payment_health_ok, "shipping": m.shipping_health_ok} ``

2. Your three actions, and nothing more. Write restart_worker, clear_cache and rotate_logs against your own tools (systemctl restart shop-worker@3, redis-cli FLUSHDB on the page cache only, logrotate -f /etc/logrotate.d/shop). Give each its input range and its interlocks. Do not write a function for anything destructive, so there is nothing to call.
3. Your rules, Playbooks and limits. Edit RULES, EXPLAIN, PLAYBOOKS and ALLOWLIST with your own service levels. Keep the standard fault codes (Part 3). Ship the Playbooks as proposed: your customer, the shop owner, adopts each one (PUT /heart/v1/tending/rules/{id}, with proof of presence), and adds its allowlist entry.
4. Your escalation route and privacy table. Change escalate() to your on-call route (pager, chat, email). Go through values() and give every value its level (Step 3 of the case study): traffic and resources internal, revenue private, customer data not in HEART at all.

Then keep monitor() as it is, store the device ID, claim code and key in the server's secret store, serve the API on loopback or behind TLS, run the HEART-Rating client and Part 9's tests, and claim only what passed.

8. Test it#

From the project's top folder, with Node 20+ and Python 3 with pip install --user "jsonschema[format]>=4.18" referencing:

python3 onlineShopSimulator/test_e2e.py

First it runs a second simulator, local only, and checks drain, graceful restart (it fails rather than force, and can be cancelled), force restart (held until the button is pressed) and every refusal, with each reply checked against the schemas. That takes about half a minute. Then it starts a HEART Hub site on this computer, runs the simulator with every staged fault at 5x speed, signs in, registers the server through the site's own form, and checks for about four minutes that every message is valid, public views hold only public data, the owner can run a Playbook's action by hand and its guard still applies, every destructive request is refused, all three Playbooks fired and stayed within maxPerDay and cooldownSec, the daily limit refuses, the Playbook whose fix did not help is suspended and escalates, uncovered problems are escalated, Stop works and restarting needs the local button, every staged fault reaches the owner's page with its fault code, no measurement, order or revenue reaches the site, and the page stays private.

Licence: Apache-2.0. Part of HEART, the open standard at heartcortex.ai.