stern-event-agent

A passive gameplay-event logger for Stern Spike 2 pinball, developed against King Kong LE 0.97. It records the game's own events with RFC 3339 UTC timestamps (millisecond precision) — the game is not modified, nothing scores, nothing blocks.

The logger is a PAD mode: a small shared object (mode.so) that the game process loads via LD_PRELOAD. It hooks function sites inside the game, and every hook is verified against the first two machine instructions of that function before it is touched — a wrong or missing port hooks nothing, logs why, and the game runs stock.

Status (as of 2026-10-12)

Part	State
Bus events (EV): game/ball/bonus start/end, tilt, all players' scores at bonus end	deployed on the machine, verified in play
Shots (SHOT): every named shot dispatched while a ball is up	deployed on the machine, verified in play
Stock modes (gamelog: MODE START/END): the game's own 22 modes with score delta	deployed on the machine, verified in play (mode_score is approximate)
Award screens (AWARD id=…): every award the game shows, extra-ball tracking	deployed on the machine, verified in play — except the extra-ball award id is not yet confirmed (below)
Switch/button log (/dump/switch.log, SW lines with generated names)	deployed on the machine, verified in play (flippers on both edges; sites located with portswitch.switch_source and added to the port)
High-score reports (HSTD): the game's own high-score table report, with all players' scores	built, deploy pending — fires when the game sends /api/v1/game/high_score_events to Stern's agent; the entries (initials) need new per-game RE (below)

The machine currently runs the v3 build (events, shots, modes, awards, switch log). The v3 work needed more than a rebuild: the port had no switch_edge / switch_drain sites or switch_* values, so the runtime's edge logger could never arm. They were located in the King Kong LE 0.97 ELF with the decryptor's pinball_decryptor.plugins.stern.portswitch.switch_source() (generation-B framework) and added to ports/king_kong_le-0.97.port.

Files

File	What it is
gamelog.c	the logger: a passive PAD mode. It never pm_begin()s, never blocks, allocates nothing — it only watches and writes [GAMELOG] lines
pad_mode_runtime.c	the PAD Modes SDK runtime, patched: (a) a GAMELOG section — passive stock-mode and award watching, exporting gl_award_seq() / gl_last_award_id() to modes; (b) the per-edge switch logger to /dump/switch.log, named by the port's switch lines; (c) the Insider Connected gate made passive; (d) multi-game port selection (port_select — below). Our exact changes vs upstream: patch/runtime-patch.diff
pad_mode.h, pad_stock.h	unmodified SDK headers (needed by the build)
build_mode.sh	unmodified SDK build script: compiles the runtime + your mode into one -nostdlib shared object and refuses banned libc calls
event_probe.c, event_read.py	the event-bus measurement instrument and its log reader — how bus event ids are identified for another game (MODE_SDK.md, "Measuring events on another game")
tests/test_multi_game.py	host-compiled tests for the port selection, the ports-directory scan and the switch log (they lift the runtime's own code verbatim; the address-dependent ones need Linux — run them in the Debian container below)
ports/king_kong_le-0.97.port	the King Kong LE 0.97 port: 121 verified sites, 55 named shots, 6 bus events. Installed on the machine as /usr/local/padmode/ports/king_kong_le-0.97.port
shipper/	logshipper: a small static Go service for the machine that tails /dump/mode.log + /dump/switch.log and pushes them to Loki — go test ./shipper, built with shipper/build.sh, run by /etc/init.d/log_shipper
k8s/loki/	the Loki the shipper pushes to: monolithic, filesystem on CephFS, 30-day retention, at https://loki.k8s-home.lqx.net, with the Grafana datasource auto-registered

Multiple games

One mode.so runs on any game that has a port. The machine says which game it is on by which port's sites verify — the runtime tries every candidate in order and keeps the first whose core functions match the running build's instruction words (the same gate that keeps a wrong port from hooking anything):

1. every *.port in /usr/local/padmode/ports/ and /dump/ports/ (name order is the directory's; only the right build's port can verify, so the order does not matter), then
2. the single files /usr/local/padmode/game.port and /dump/game.port, the old one-game layout.

The boot log says what happened:

[pad] port /usr/local/padmode/ports/godzilla_le-1.16.port: says godzilla_le 1.16 - not this game's port, skipped
[pad] port /usr/local/padmode/ports/king_kong_le-0.97.port: king_kong_le 0.97, 121 sites, 55 shots

A game update changes the words, so its old port stops matching — the machine logs the refusal and runs stock until the new game version's port is deployed. The switch log's buttons and their names come from the port's switch lines; a port that names none logs no switch edges. | patch/runtime-patch.diff | this project's changes to the upstream runtime (base: pinball-asset-decryptor commit 7faf8f08, the last upstream change to that file) |

What gets logged

/dump/mode.log (deployed)

Append-only, RFC 3339 UTC stamps with millisecond precision, survives reboots:

[pad] port ...: king_kong_le 0.97, 121 sites, 55 shots          <- runtime armed (all hook words verified)
[GAMELOG] 2026-10-12T20:45:12.123Z up: game king_kong_le 0.97, 55 named shots; event ids 77/37/52/40/200/198; extra_ball_award=13
[GAMELOG] 2026-10-12T20:45:12.123Z EV game_start - GAME START
[GAMELOG] 2026-10-12T20:45:12.123Z EV ball_start player=1 ball=1 score=0
[GAMELOG] 2026-10-12T20:45:12.123Z SHOT X1 player=1 ball=1 score=7,349,830    <- all 55 named shots
[GAMELOG] 2026-10-12T20:45:12.123Z EV ball_end player=1 ball=1 score=91,153,090
[GAMELOG] 2026-10-12T20:45:12.123Z EV bonus_end
[GAMELOG] 2026-10-12T20:45:12.123Z SCORES P1=95,421,190 P2=0 P3=0 P4=0
[GAMELOG] 2026-10-12T20:45:12.123Z EV tilt_warning player=1
[GAMELOG] 2026-10-12T20:45:12.123Z EV tilt player=1 ball=2
[pad] gamelog: 2026-10-12T20:45:12.123Z MODE START 5 (Island Scene Escape the Swamp) player=1 score=...
[pad] gamelog: 2026-10-12T20:45:12.123Z MODE END 5 (Island Scene Escape the Swamp) player=1 score=... mode_score=... dur=...ms
[GAMELOG] 2026-10-12T20:45:12.123Z AWARD EXTRA BALL (id=13) player=1              <- next ball logs as "ball N (EXTRA BALL)"
[GAMELOG] 2026-10-12T20:45:12.123Z AWARD id=<n> player=1 score=...                <- every award screen
[GAMELOG] 2026-10-12T20:45:12.123Z HSTD high-score report - P1=95,421,190 P2=0 P3=0 P4=0

Bus event ids (King Kong 0.97): game_start=0x4d, ball_start=0x25, ball_end=0x34, bonus_end=0x28, tilt_warning=0xc8, tilt=0xc6. This build has no bus id for game over (the last ball_end/bonus_end is the close) or for bonus start.

Extra balls: the game re-fires ball_start for an extra ball. The runtime's award-screen hook counts the awards, and one whose id matches extra_ball_award (default 13, overridable with a value extra_ball_award line in the port file) makes the replayed ball log as ball N (EXTRA BALL) instead of advancing the count.

High scores: when the game builds its high-score table report for Stern's agent (/api/v1/game/high_score_events — the endpoint the Insider gate sits on), the logger writes an HSTD line with all four players' scores. This is the report the game sends when a game has changed the table, so an HSTD line marks a high-score event without any new per-game RE; the entries themselves (initials, position) live deeper in the report and are not captured yet.

Mode detail: the runtime polls each of the game's 22 mode objects' ACTIVE state per tick and logs START/END with the player's score at each edge — mode_score= is the score delta across the mode (approximate when other scoring runs simultaneously).

Insider Connected: stock. The upstream runtime vetoes score/session reports whenever modes are installed; this build makes that gate passive — reports flow exactly as stock (and are logged once), and the logger adds no scores of its own. Known cosmetic issue: the boot banner still prints the upstream "score reports stay on this machine" line — the later passive gate line is the truth.

/dump/switch.log (deployed)

The runtime's switch-edge handler writes every edge of the human-activated buttons to /dump/switch.log — the lockdown/action button (34), the start button (36) and the two flipper buttons (62/63), attract included. Names are quoted; the upper-flipper button switches (64/65) are wired in parallel with the main flipper buttons on this cabinet, so they are left out:

2026-10-12T20:45:12.123Z SW <id> "<NAME>" level=<0|1> hit=<0|1> game=<0|1>

level is the switch's raw state after the edge, hit is the game's own "counts as a hit" verdict, game whether a player is up. The logged ids and names are the port's switch lines (on King Kong: 34 lockdown, 36 start, 62/63 the flipper buttons).

Build

Cross-compile on macOS in a Debian container (armhf, hard-float, -nostdlib):

docker run --rm --platform linux/arm64 -v "$PWD:/work" -w /work debian:12 \
  sh -c 'apt-get update -qq && apt-get install -y -qq gcc-arm-linux-gnueabihf >/dev/null \
  && bash build_mode.sh -o mode.so gamelog.c'

build_mode.sh compiles pad_mode_runtime.c + gamelog.c into one object (~228 KB) and fails the build if any pm_ symbol does not exist or any banned libc call is made (allocation, threads, sleeping, stdio, processes). The few libc calls that are used (clock_gettime, open, write, close, …) resolve from the game's own libc when the object loads.

The event probe builds without the runtime: build_mode.sh -p -o event_probe.so event_probe.c

The port-selection tests run on any Linux host (the address-dependent ones skip on macOS, whose arm64 maps nothing below 4 GiB):

docker run --rm --platform linux/arm64 -v "$PWD:/work" -w /work debian:12 \
  sh -c 'apt-get update -qq && apt-get install -y -qq python3 python3-pytest gcc >/dev/null \
  && python3 -m pytest tests/test_multi_game.py -q'

Install / deploy (the machine)

The machine (hostname stern-spike2) has root SSH (key-only, port 22). The logger lives on the rootfs, which is normally read-only:

Path on the machine	From this repo
/usr/local/padmode/mode.so	the built mode.so
/usr/local/padmode/ports/king_kong_le-0.97.port	ports/king_kong_le-0.97.port (one file per game; see "Multiple games")
/etc/init.d/game_monitor	boot hook: preloads mode.so into the game only if it exists (stock script kept at game_monitor.stock)

Deploy a new build:

scp mode.so root@stern-spike2:/tmp/
ssh root@stern-spike2 'mount -o remount,rw /; cp /tmp/mode.so /usr/local/padmode/mode.so;
  chmod 755 /usr/local/padmode/mode.so; sync; mount -o remount,ro / || true; reboot'

Live view: ssh root@stern-spike2 'tail -f /dump/mode.log'

Housekeeping: /dump is a 532 MiB rw partition shared with crash dumps and connectivity logs; mode.log grows a few MB per long session (a few hundred SHOT lines per ball is normal). Pull it over SSH and rotate as needed; : > /dump/mode.log truncates it safely while running (same for switch.log once deployed).

Rollback: rm -rf /usr/local/padmode, restore /etc/init.d/game_monitor.stock, reboot — the machine is stock again (SSH remains).

Measuring events for another game

event_probe.c hooks the game's event bus (subscribe / dispatch) and its tick, and logs every registration and per-window dispatch counts to /dump/event.log. Procedure (the full version is MODE_SDK.md, "Measuring events on another game"):

1. build_mode.sh -p -o event_probe.so event_probe.c
2. Put the target's subscribe / dispatch / tick sites (address + first two instruction words) in /dump/event.sites — the SDK's port_words.py (in the decryptor repo) fills and checks those words from the game's ELF.
3. Preload event_probe.so with PAD_MODE_SO (beside the stock runtime — its trampoline keeps the moved words where the runtime still verifies them).
4. Play; at interesting moments echo <label> > /dump/event.mark to close a window (and open the next); echo <id> > /dump/event.force starts/stops a stock mode the way a rule does.
5. event_read.py /dump/event.log — which ids fired in which window, with handlers, priorities and callers.

The ids you name that way become the event lines of a port (the six in king_kong_le-0.97.port were found this way, in the SDK's emulator).

Log aggregation (Loki)

The machine ships its own logs: logshipper (in shipper/) tails /dump/mode.log and /dump/switch.log and pushes them to Loki at https://loki.k8s-home.lqx.net (deployed from k8s/loki/ on the home cluster, Grafana datasource auto-registered). Query in Grafana or logcli:

{job="pinball", source="mode.log"} |= "ball_start"    # every ball start
{job="pinball", source="mode.log"} |= "MODE START"    # game modes
{job="pinball", source="switch.log"}                  # button edges

Labels: job=pinball, machine, source (mode.log / switch.log), and game (from the logger's boot line, e.g. king_kong_le). Each line keeps its own RFC 3339 stamp as the Loki timestamp, so live tailing in Grafana is the game, in real time, with the machine's own clock.

Shipper details worth knowing:

• Per-file read offsets live in /dump/.logshipper.state and are saved only after a successful push — a down Loki means retry-then-catch-up, never loss.
• First run starts at each file's end; run once with SHIP_FROM_START=1 to ship the history already on disk (Loki accepts entries up to a year old, and the shipper calls /flush after a backfill — old entries are invisible in the ingester until they reach the store).
• Truncation (: > /dump/mode.log) is noticed and the file restarts from 0.
• Config is env (LOKI_URL, SHIP_FILES, SHIP_STATE, MACHINE, SHIP_FROM_START, SHIP_INSECURE, SHIP_EVERY); defaults match the machine.

Deploy the shipper:

cd shipper && ./build.sh
scp logshipper root@stern-spike2:/tmp/
ssh root@stern-spike2 'mount -o remount,rw /; cp /tmp/logshipper /usr/local/bin/;
  cp /tmp/log_shipper /etc/init.d/ && chmod 755 /usr/local/bin/logshipper /etc/init.d/log_shipper;
  ln -sf ../init.d/log_shipper /etc/rc5.d/S60log_shipper; sync; mount -o remount,ro / || true;
  /etc/init.d/log_shipper start'

Unfinished work / known issues

• The extra-ball award id is unconfirmed. Default 13 is a static guess from the game's object-name tables; every award id is logged, so earn an extra ball, read the AWARD line, and pin it down (or override it in the port).
• The master SD image does not contain the logger or the shipper — they were installed over SSH onto the card in the machine. Making them survive a re-flash needs a debugfs inject of mode.so + the ports + the hooked game_monitor + logshipper into the master image.
• Cosmetic: the boot banner still prints the upstream "score reports stay on this machine" line (see Insider Connected above).
• mode_score is an approximation (see above).

Provenance

Built on the PAD Modes SDK (pinball-asset-decryptor/tools/spike2_emu/modes/sdk/ — the mattlqx/pinball-asset-decryptor Gitea repo, itself a fork of davidvanderburgh/pinball-asset-decryptor; the SDK docs there — MODE_SDK.md, MODE_PARAMETERS.md, MODE_LIMITS.md — apply to everything in this repo). Everything in this repo is unmodified SDK content except:

• gamelog.c (new)
• pad_mode_runtime.c (upstream runtime + the patch in patch/runtime-patch.diff)
• tests/test_multi_game.py (new)
• README.md

ports/king_kong_le-0.97.port, event_probe.c, event_read.py, build_mode.sh and the headers are the SDK's, unmodified (the port is emulator-proven by the SDK repo authors).

The machine's SSH setup (dropbear + sftp-server on the Spike 2) is documented separately in the local pinball-assets/README.md and is not part of this project — this logger uses that SSH, it does not provide it.