The gameplay logs sat on /dump on the machine, readable only over SSH. Now
they land in Loki for query and tail:
- k8s/loki/loki.yaml: Loki 3.4.4 monolithic on the home cluster's monitoring
namespace — filesystem storage on CephFS, 30-day retention, traefik ingress
at loki.k8s-home.lqx.net with a cert-manager cert, Grafana datasource
auto-registered via the sidecar. reject_old_samples_max_age is a year:
the machines backfill their history on first ship.
- shipper/: logshipper, a ~300-line static Go service for the machine. Tails
/dump/mode.log and /dump/switch.log, batches, pushes to Loki's push API.
Per-file offsets are saved only after a successful push (down Loki =
retry-then-catch-up, never loss); first run starts at each file's end,
SHIP_FROM_START=1 ships history; truncation is noticed. Lines keep their
own RFC 3339 stamp as the Loki timestamp; unstamped lines (the boot
banner) inherit the next stamped line's, so a stream never starts at
"now" and jumps back. After a backfill it calls /flush — old entries are
invisible in the ingester's head until they reach the store (found the
hard way: Loki answers 204 to an old entry on a new stream and then never
serves it). Labels: job=pinball, machine, source, game (from the boot
line). /etc/init.d/log_shipper runs it at boot.
Deployed: Loki is live and verified (query through the ingress). The shipper
is verified end-to-end against the cluster from this Mac; the machine was
unreachable at deploy time — the README has the install commands.
Co-Authored-By: Claude Code <noreply@anthropic.com>
