feat: log aggregation — the machine ships its logs to Loki
The gameplay logs sat on /dump on the machine, readable only over SSH. Now they land in Loki for query and tail: - k8s/loki/loki.yaml: Loki 3.4.4 monolithic on the home cluster's monitoring namespace — filesystem storage on CephFS, 30-day retention, traefik ingress at loki.k8s-home.lqx.net with a cert-manager cert, Grafana datasource auto-registered via the sidecar. reject_old_samples_max_age is a year: the machines backfill their history on first ship. - shipper/: logshipper, a ~300-line static Go service for the machine. Tails /dump/mode.log and /dump/switch.log, batches, pushes to Loki's push API. Per-file offsets are saved only after a successful push (down Loki = retry-then-catch-up, never loss); first run starts at each file's end, SHIP_FROM_START=1 ships history; truncation is noticed. Lines keep their own RFC 3339 stamp as the Loki timestamp; unstamped lines (the boot banner) inherit the next stamped line's, so a stream never starts at "now" and jumps back. After a backfill it calls /flush — old entries are invisible in the ingester's head until they reach the store (found the hard way: Loki answers 204 to an old entry on a new stream and then never serves it). Labels: job=pinball, machine, source, game (from the boot line). /etc/init.d/log_shipper runs it at boot. Deployed: Loki is live and verified (query through the ingress). The shipper is verified end-to-end against the cluster from this Mac; the machine was unreachable at deploy time — the README has the install commands. Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
Executable
+21
@@ -0,0 +1,21 @@
|
||||
#!/bin/sh
|
||||
# log_shipper - tail /dump/*.log and push them to Loki (see shipper/README.md).
|
||||
# Config comes from the environment; defaults in the binary match the machine.
|
||||
|
||||
case "$1" in
|
||||
start)
|
||||
start-stop-daemon -S -b -m -p /run/logshipper.pid -x /usr/local/bin/logshipper
|
||||
;;
|
||||
stop)
|
||||
start-stop-daemon -K -p /run/logshipper.pid
|
||||
;;
|
||||
restart)
|
||||
"$0" stop
|
||||
sleep 1
|
||||
"$0" start
|
||||
;;
|
||||
*)
|
||||
echo "usage: $0 {start|stop|restart}"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
Reference in New Issue
Block a user