stern-event-agent/README.md

# stern-event-agent

A passive gameplay-event logger for Stern Spike 2 pinball, developed against
**King Kong LE 0.97**. It records the game's own events with RFC 3339 UTC
timestamps (millisecond precision) — the game is not modified, nothing scores,
nothing blocks.

The logger is a PAD mode: a small shared object (`mode.so`) that the game
process loads via `LD_PRELOAD`. It hooks function sites inside the game, and
every hook is verified against the first two machine instructions of that
function before it is touched — a wrong or missing port hooks nothing, logs
why, and the game runs stock.

## Status (as of 2026-10-12)

| Part | State |
|---|---|
| Bus events (`EV`): game/ball/bonus start/end, tilt, all players' scores at bonus end | **deployed on the machine, verified in play** |
| Shots (`SHOT`): every named shot dispatched while a ball is up | **deployed on the machine, verified in play** |
| Stock modes (`gamelog: MODE START/END`): the game's own 22 modes with score delta | **deployed on the machine, verified in play** (`mode_score` is approximate) |
| Award screens (`AWARD id=…`): every award the game shows, extra-ball tracking | **deployed on the machine, verified in play** — except the extra-ball award id is not yet confirmed (below) |
| Switch/button log (`/dump/switch.log`, `SW` lines with generated names) | **deployed on the machine, verified in play** (flippers on both edges; sites located with `portswitch.switch_source` and added to the port) |

The machine currently runs the v3 build (events, shots, modes, awards, switch
log). The v3 work needed more than a rebuild: the port had no `switch_edge` /
`switch_drain` sites or `switch_*` values, so the runtime's edge logger could
never arm. They were located in the King Kong LE 0.97 ELF with the decryptor's
`pinball_decryptor.plugins.stern.portswitch.switch_source()` (generation-B
framework) and added to `ports/king_kong_le-0.97.port`.

## Files

| File | What it is |
|---|---|
| `gamelog.c` | the logger: a passive PAD mode. It never `pm_begin()s`, never blocks, allocates nothing — it only watches and writes `[GAMELOG]` lines |
| `pad_mode_runtime.c` | the PAD Modes SDK runtime, **patched**: (a) a GAMELOG section — passive stock-mode and award watching, exporting `gl_award_seq()` / `gl_last_award_id()` to modes; (b) the per-edge switch logger to `/dump/switch.log`, named by the port's `switch` lines; (c) the Insider Connected gate made passive; (d) multi-game port selection (`port_select` — below). Our exact changes vs upstream: `docs/runtime-patch.diff` |
| `pad_mode.h`, `pad_stock.h` | unmodified SDK headers (needed by the build) |
| `build_mode.sh` | unmodified SDK build script: compiles the runtime + your mode into one `-nostdlib` shared object and refuses banned libc calls |
| `event_probe.c`, `event_read.py` | the event-bus measurement instrument and its log reader — how bus event ids are identified for another game (MODE_SDK.md, "Measuring events on another game") |
| `tests/test_multi_game.py` | host-compiled tests for the port selection, the ports-directory scan and the switch log (they lift the runtime's own code verbatim; the address-dependent ones need Linux — run them in the Debian container below) |
| `ports/king_kong_le-0.97.port` | the King Kong LE 0.97 port: 121 verified sites, 55 named shots, 6 bus events. Installed on the machine as `/usr/local/padmode/ports/king_kong_le-0.97.port` |
| `shipper/` | `logshipper`: a small static Go service for the machine that tails `/dump/mode.log` + `/dump/switch.log` and pushes them to Loki — `go test ./shipper`, built with `shipper/build.sh`, run by `/etc/init.d/log_shipper` |
| `k8s/loki/` | the Loki the shipper pushes to: monolithic, filesystem on CephFS, 30-day retention, at `https://loki.k8s-home.lqx.net`, with the Grafana datasource auto-registered |

## Multiple games

One `mode.so` runs on any game that has a port. The machine says which game it is
on by which port's sites verify — the runtime tries every candidate in order and
keeps the first whose core functions match the running build's instruction words
(the same gate that keeps a wrong port from hooking anything):

1. every `*.port` in `/usr/local/padmode/ports/` and `/dump/ports/` (name order
   is the directory's; only the right build's port can verify, so the order
   does not matter), then
2. the single files `/usr/local/padmode/game.port` and `/dump/game.port`, the
   old one-game layout.

The boot log says what happened:

```
[pad] port /usr/local/padmode/ports/godzilla_le-1.16.port: says godzilla_le 1.16 - not this game's port, skipped
[pad] port /usr/local/padmode/ports/king_kong_le-0.97.port: king_kong_le 0.97, 121 sites, 55 shots
```

A game update changes the words, so its old port stops matching — the machine
logs the refusal and runs stock until the new game version's port is deployed.
The switch log's buttons and their names come from the port's `switch` lines;
a port that names none logs no switch edges.
| `docs/runtime-patch.diff` | this project's changes to the upstream runtime (base: `pinball-asset-decryptor` commit `7faf8f08`, the last upstream change to that file) |

## What gets logged

### `/dump/mode.log` (deployed)

Append-only, RFC 3339 UTC stamps with millisecond precision, survives
reboots:

```
[pad] port ...: king_kong_le 0.97, 121 sites, 55 shots          <- runtime armed (all hook words verified)
[GAMELOG] 2026-10-12T20:45:12.123Z up: game king_kong_le 0.97, 55 named shots; event ids 77/37/52/40/200/198; extra_ball_award=13
[GAMELOG] 2026-10-12T20:45:12.123Z EV game_start - GAME START
[GAMELOG] 2026-10-12T20:45:12.123Z EV ball_start player=1 ball=1 score=0
[GAMELOG] 2026-10-12T20:45:12.123Z SHOT X1 player=1 ball=1 score=7,349,830    <- all 55 named shots
[GAMELOG] 2026-10-12T20:45:12.123Z EV ball_end player=1 ball=1 score=91,153,090
[GAMELOG] 2026-10-12T20:45:12.123Z EV bonus_end
[GAMELOG] 2026-10-12T20:45:12.123Z SCORES P1=95,421,190 P2=0 P3=0 P4=0
[GAMELOG] 2026-10-12T20:45:12.123Z EV tilt_warning player=1
[GAMELOG] 2026-10-12T20:45:12.123Z EV tilt player=1 ball=2
[pad] gamelog: 2026-10-12T20:45:12.123Z MODE START 5 (Island Scene Escape the Swamp) player=1 score=...
[pad] gamelog: 2026-10-12T20:45:12.123Z MODE END 5 (Island Scene Escape the Swamp) player=1 score=... mode_score=... dur=...ms
[GAMELOG] 2026-10-12T20:45:12.123Z AWARD EXTRA BALL (id=13) player=1              <- next ball logs as "ball N (EXTRA BALL)"
[GAMELOG] 2026-10-12T20:45:12.123Z AWARD id=<n> player=1 score=...                <- every award screen
```

Bus event ids (King Kong 0.97): `game_start`=0x4d, `ball_start`=0x25,
`ball_end`=0x34, `bonus_end`=0x28, `tilt_warning`=0xc8, `tilt`=0xc6. This build
has **no** bus id for game over (the last `ball_end`/`bonus_end` is the close)
or for bonus start.

**Extra balls**: the game re-fires `ball_start` for an extra ball. The runtime's
award-screen hook counts the awards, and one whose id matches
`extra_ball_award` (default **13**, overridable with a `value extra_ball_award`
line in the port file) makes the replayed ball log as `ball N (EXTRA BALL)`
instead of advancing the count.

**Mode detail**: the runtime polls each of the game's 22 mode objects' ACTIVE
state per tick and logs START/END with the player's score at each edge —
`mode_score=` is the score delta across the mode (approximate when other
scoring runs simultaneously).

**Insider Connected**: stock. The upstream runtime vetoes score/session reports
whenever modes are installed; this build makes that gate **passive** — reports
flow exactly as stock (and are logged once), and the logger adds no scores of
its own. Known cosmetic issue: the boot banner still prints the upstream "score
reports stay on this machine" line — the later `passive gate` line is the truth.

### `/dump/switch.log` (deployed)

The runtime's switch-edge handler writes every edge of the human-activated
buttons to `/dump/switch.log` — the lockdown/action button (34), the start
button (36) and the two flipper buttons (62/63), attract included. Names are
quoted; the upper-flipper button switches (64/65) are wired in parallel with
the main flipper buttons on this cabinet, so they are left out:

```
2026-10-12T20:45:12.123Z SW <id> "<NAME>" level=<0|1> hit=<0|1> game=<0|1>
```

`level` is the switch's raw state after the edge, `hit` is the game's own
"counts as a hit" verdict, `game` whether a player is up. The logged ids and
names are the port's `switch` lines (on King Kong: 34 lockdown, 36 start,
62/63 the flipper buttons).

## Build

Cross-compile on macOS in a Debian container (armhf, hard-float, `-nostdlib`):

```bash
docker run --rm --platform linux/arm64 -v "$PWD:/work" -w /work debian:12 \
  sh -c 'apt-get update -qq && apt-get install -y -qq gcc-arm-linux-gnueabihf >/dev/null \
  && bash build_mode.sh -o mode.so gamelog.c'
```

`build_mode.sh` compiles `pad_mode_runtime.c` + `gamelog.c` into one object
(~228 KB) and fails the build if any `pm_` symbol does not exist or any banned
libc call is made (allocation, threads, sleeping, stdio, processes). The few
libc calls that are used (`clock_gettime`, `open`, `write`, `close`, …) resolve
from the game's own libc when the object loads.

The event probe builds without the runtime:
`build_mode.sh -p -o event_probe.so event_probe.c`

The port-selection tests run on any Linux host (the address-dependent ones skip
on macOS, whose arm64 maps nothing below 4 GiB):

```bash
docker run --rm --platform linux/arm64 -v "$PWD:/work" -w /work debian:12 \
  sh -c 'apt-get update -qq && apt-get install -y -qq python3 python3-pytest gcc >/dev/null \
  && python3 -m pytest tests/test_multi_game.py -q'
```

## Install / deploy (the machine)

The machine (hostname `stern-spike2`) has root SSH (key-only, port 22). The
logger lives on the rootfs, which is normally read-only:

| Path on the machine | From this repo |
|---|---|
| `/usr/local/padmode/mode.so` | the built `mode.so` |
| `/usr/local/padmode/ports/king_kong_le-0.97.port` | `ports/king_kong_le-0.97.port` (one file per game; see "Multiple games") |
| `/etc/init.d/game_monitor` | boot hook: preloads `mode.so` into the game **only if it exists** (stock script kept at `game_monitor.stock`) |

Deploy a new build:

```bash
scp mode.so root@stern-spike2:/tmp/
ssh root@stern-spike2 'mount -o remount,rw /; cp /tmp/mode.so /usr/local/padmode/mode.so;
  chmod 755 /usr/local/padmode/mode.so; sync; mount -o remount,ro / || true; reboot'
```

Live view: `ssh root@stern-spike2 'tail -f /dump/mode.log'`

Housekeeping: `/dump` is a 532 MiB rw partition shared with crash dumps and
connectivity logs; `mode.log` grows a few MB per long session (a few hundred
`SHOT` lines per ball is normal). Pull it over SSH and rotate as needed;
`: > /dump/mode.log` truncates it safely while running (same for `switch.log`
once deployed).

Rollback: `rm -rf /usr/local/padmode`, restore `/etc/init.d/game_monitor.stock`,
reboot — the machine is stock again (SSH remains).

## Measuring events for another game

`event_probe.c` hooks the game's event bus (`subscribe` / `dispatch`) and its
tick, and logs every registration and per-window dispatch counts to
`/dump/event.log`. Procedure (the full version is MODE_SDK.md, "Measuring
events on another game"):

1. `build_mode.sh -p -o event_probe.so event_probe.c`
2. Put the target's `subscribe` / `dispatch` / `tick` sites (address + first two
   instruction words) in `/dump/event.sites` — the SDK's `port_words.py`
   (in the decryptor repo) fills and checks those words from the game's ELF.
3. Preload `event_probe.so` with `PAD_MODE_SO` (beside the stock runtime — its
   trampoline keeps the moved words where the runtime still verifies them).
4. Play; at interesting moments `echo <label> > /dump/event.mark` to close a
   window (and open the next); `echo <id> > /dump/event.force` starts/stops a
   stock mode the way a rule does.
5. `event_read.py /dump/event.log` — which ids fired in which window, with
   handlers, priorities and callers.

The ids you name that way become the `event` lines of a port (the six in
`king_kong_le-0.97.port` were found this way, in the SDK's emulator).

## Log aggregation (Loki)

The machine ships its own logs: `logshipper` (in `shipper/`) tails
`/dump/mode.log` and `/dump/switch.log` and pushes them to Loki at
`https://loki.k8s-home.lqx.net` (deployed from `k8s/loki/` on the home
cluster, Grafana datasource auto-registered). Query in Grafana or logcli:

```
{job="pinball", source="mode.log"} |= "ball_start"    # every ball start
{job="pinball", source="mode.log"} |= "MODE START"    # game modes
{job="pinball", source="switch.log"}                  # button edges
```

Labels: `job=pinball`, `machine`, `source` (`mode.log` / `switch.log`), and
`game` (from the logger's boot line, e.g. `king_kong_le`). Each line keeps its
own RFC 3339 stamp as the Loki timestamp, so live tailing in Grafana is the
game, in real time, with the machine's own clock.

Shipper details worth knowing:

- Per-file read offsets live in `/dump/.logshipper.state` and are saved only
  after a successful push — a down Loki means retry-then-catch-up, never loss.
- First run starts at each file's **end**; run once with `SHIP_FROM_START=1`
  to ship the history already on disk (Loki accepts entries up to a year old,
  and the shipper calls `/flush` after a backfill — old entries are invisible
  in the ingester until they reach the store).
- Truncation (`: > /dump/mode.log`) is noticed and the file restarts from 0.
- Config is env (`LOKI_URL`, `SHIP_FILES`, `SHIP_STATE`, `MACHINE`,
  `SHIP_FROM_START`, `SHIP_INSECURE`, `SHIP_EVERY`); defaults match the
  machine.

Deploy the shipper:

```bash
cd shipper && ./build.sh
scp logshipper root@stern-spike2:/tmp/
ssh root@stern-spike2 'mount -o remount,rw /; cp /tmp/logshipper /usr/local/bin/;
  cp /tmp/log_shipper /etc/init.d/ && chmod 755 /usr/local/bin/logshipper /etc/init.d/log_shipper;
  ln -sf ../init.d/log_shipper /etc/rc5.d/S60log_shipper; sync; mount -o remount,ro / || true;
  /etc/init.d/log_shipper start'
```

## Unfinished work / known issues

- **The extra-ball award id is unconfirmed.** Default 13 is a static guess from
  the game's object-name tables; every award id is logged, so earn an extra
  ball, read the `AWARD` line, and pin it down (or override it in the port).
- **The master SD image does not contain the logger or the shipper** — they
  were installed over SSH onto the card in the machine. Making them survive a
  re-flash needs a debugfs inject of `mode.so` + the ports + the hooked
  `game_monitor` + `logshipper` into the master image.
- **Cosmetic**: the boot banner still prints the upstream "score reports stay
  on this machine" line (see Insider Connected above).
- `mode_score` is an approximation (see above).

## Provenance

Built on the PAD Modes SDK
(`pinball-asset-decryptor/tools/spike2_emu/modes/sdk/` — the
`mattlqx/pinball-asset-decryptor` Gitea repo, itself a fork of
`davidvanderburgh/pinball-asset-decryptor`; the SDK docs there — `MODE_SDK.md`,
`MODE_PARAMETERS.md`, `MODE_LIMITS.md` — apply to everything in this repo).
Everything in this repo is unmodified SDK content except:

- `gamelog.c` (new)
- `pad_mode_runtime.c` (upstream runtime + the patch in `docs/runtime-patch.diff`)
- `tests/test_multi_game.py` (new)
- `README.md`

`ports/king_kong_le-0.97.port`, `event_probe.c`, `event_read.py`,
`build_mode.sh` and the headers are the SDK's, unmodified (the port is
emulator-proven by the SDK repo authors).

The machine's SSH setup (dropbear + sftp-server on the Spike 2) is documented
separately in the local `pinball-assets/README.md` and is **not** part of this
project — this logger uses that SSH, it does not provide it.