feat: log aggregation — the machine ships its logs to Loki

The gameplay logs sat on /dump on the machine, readable only over SSH. Now
they land in Loki for query and tail:

- k8s/loki/loki.yaml: Loki 3.4.4 monolithic on the home cluster's monitoring
  namespace — filesystem storage on CephFS, 30-day retention, traefik ingress
  at loki.k8s-home.lqx.net with a cert-manager cert, Grafana datasource
  auto-registered via the sidecar. reject_old_samples_max_age is a year:
  the machines backfill their history on first ship.
- shipper/: logshipper, a ~300-line static Go service for the machine. Tails
  /dump/mode.log and /dump/switch.log, batches, pushes to Loki's push API.
  Per-file offsets are saved only after a successful push (down Loki =
  retry-then-catch-up, never loss); first run starts at each file's end,
  SHIP_FROM_START=1 ships history; truncation is noticed. Lines keep their
  own RFC 3339 stamp as the Loki timestamp; unstamped lines (the boot
  banner) inherit the next stamped line's, so a stream never starts at
  "now" and jumps back. After a backfill it calls /flush — old entries are
  invisible in the ingester's head until they reach the store (found the
  hard way: Loki answers 204 to an old entry on a new stream and then never
  serves it). Labels: job=pinball, machine, source, game (from the boot
  line). /etc/init.d/log_shipper runs it at boot.

Deployed: Loki is live and verified (query through the ingress). The shipper
is verified end-to-end against the cluster from this Mac; the machine was
unreachable at deploy time — the README has the install commands.

Co-Authored-By: Claude Code <noreply@anthropic.com>

README.md

@@ -39,6 +39,8 @@ framework) and added to `ports/king_kong_le-0.97.port`.
 | `event_probe.c`, `event_read.py` | the event-bus measurement instrument and its log reader — how bus event ids are identified for another game (MODE_SDK.md, "Measuring events on another game") |
 | `tests/test_multi_game.py` | host-compiled tests for the port selection, the ports-directory scan and the switch log (they lift the runtime's own code verbatim; the address-dependent ones need Linux — run them in the Debian container below) |
 | `ports/king_kong_le-0.97.port` | the King Kong LE 0.97 port: 121 verified sites, 55 named shots, 6 bus events. Installed on the machine as `/usr/local/padmode/ports/king_kong_le-0.97.port` |
+| `shipper/` | `logshipper`: a small static Go service for the machine that tails `/dump/mode.log` + `/dump/switch.log` and pushes them to Loki — `go test ./shipper`, built with `shipper/build.sh`, run by `/etc/init.d/log_shipper` |
+| `k8s/loki/` | the Loki the shipper pushes to: monolithic, filesystem on CephFS, 30-day retention, at `https://loki.k8s-home.lqx.net`, with the Grafana datasource auto-registered |
 
 ## Multiple games
 
@@ -209,17 +211,57 @@ events on another game"):
 The ids you name that way become the `event` lines of a port (the six in
 `king_kong_le-0.97.port` were found this way, in the SDK's emulator).
 
+## Log aggregation (Loki)
+
+The machine ships its own logs: `logshipper` (in `shipper/`) tails
+`/dump/mode.log` and `/dump/switch.log` and pushes them to Loki at
+`https://loki.k8s-home.lqx.net` (deployed from `k8s/loki/` on the home
+cluster, Grafana datasource auto-registered). Query in Grafana or logcli:
+
+```
+{job="pinball", source="mode.log"} |= "ball_start"    # every ball start
+{job="pinball", source="mode.log"} |= "MODE START"    # game modes
+{job="pinball", source="switch.log"}                  # button edges
+```
+
+Labels: `job=pinball`, `machine`, `source` (`mode.log` / `switch.log`), and
+`game` (from the logger's boot line, e.g. `king_kong_le`). Each line keeps its
+own RFC 3339 stamp as the Loki timestamp, so live tailing in Grafana is the
+game, in real time, with the machine's own clock.
+
+Shipper details worth knowing:
+
+- Per-file read offsets live in `/dump/.logshipper.state` and are saved only
+  after a successful push — a down Loki means retry-then-catch-up, never loss.
+- First run starts at each file's **end**; run once with `SHIP_FROM_START=1`
+  to ship the history already on disk (Loki accepts entries up to a year old,
+  and the shipper calls `/flush` after a backfill — old entries are invisible
+  in the ingester until they reach the store).
+- Truncation (`: > /dump/mode.log`) is noticed and the file restarts from 0.
+- Config is env (`LOKI_URL`, `SHIP_FILES`, `SHIP_STATE`, `MACHINE`,
+  `SHIP_FROM_START`, `SHIP_INSECURE`, `SHIP_EVERY`); defaults match the
+  machine.
+
+Deploy the shipper:
+
+```bash
+cd shipper && ./build.sh
+scp logshipper root@stern-spike2:/tmp/
+ssh root@stern-spike2 'mount -o remount,rw /; cp /tmp/logshipper /usr/local/bin/;
+  cp /tmp/log_shipper /etc/init.d/ && chmod 755 /usr/local/bin/logshipper /etc/init.d/log_shipper;
+  ln -sf ../init.d/log_shipper /etc/rc5.d/S60log_shipper; sync; mount -o remount,ro / || true;
+  /etc/init.d/log_shipper start'
+```
+
 ## Unfinished work / known issues
 
 - **The extra-ball award id is unconfirmed.** Default 13 is a static guess from
   the game's object-name tables; every award id is logged, so earn an extra
   ball, read the `AWARD` line, and pin it down (or override it in the port).
-- **The master SD image does not contain the logger** — it was installed over
-  SSH onto the card in the machine. Making it survive a re-flash needs a
-  debugfs inject of `mode.so` + `game.port` + the hooked `game_monitor` into
-  the master image.
-- **No network streaming yet.** Logs are pulled over SSH; the plan is for the
-  mode to ship them itself (`socket()` is not on the build's banned list).
+- **The master SD image does not contain the logger or the shipper** — they
+  were installed over SSH onto the card in the machine. Making them survive a
+  re-flash needs a debugfs inject of `mode.so` + the ports + the hooked
+  `game_monitor` + `logshipper` into the master image.
 - **Cosmetic**: the boot banner still prints the upstream "score reports stay
   on this machine" line (see Insider Connected above).
 - `mode_score` is an approximation (see above).