feat(phase 1): media-source conversion + Streamlink supervisor + converter (1.1, 1.2, 1.3) #19
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/phase-1-media-source"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Phase 1 of the iter-3.4 stream-a-thon plan: replace OBS browser_source inputs with ffmpeg_source backed by a Streamlink supervisor delivering MPEG-TS over local UDP. Tasks 1.1, 1.2, and 1.3 are done at the unit-test layer; 1.4 (load test) and [PROD-HOST] integration steps are out of scope for this PR — see Test Plan.
Phase 1.1 — createStreamGroupV2
lib/streamInputConfig.js— purebuildStreamInputConfig+buildFfmpegSourceSettings+createStreamInputhelpers. Decoupled from the obs-websocket-js ESM module so the choreography is fully unit-testable (no jest transform issues).createStreamGroupV2(groupName, streamName, teamName, url, opts)inlib/obsClient.js, parallel to V1. DefaultuseFfmpegSource: falsepreserves V1 behaviour; flip per-call. V1 untouched as the rollback target.Phase 1.2 — Streamlink supervisor
scripts/streamlink-supervisor/:RestartTracker— 3-in-30s escalation policyPortAllocator— sequential UDP ports from a base, reuse-on-releasecommands.ts— pure CLI builders forstreamlink --stdout | ffmpeg -c copy -f mpegts udp://127.0.0.1:<port>StreamPipeline— owns one streamlink+ffmpeg pair; spawn injectedSupervisor— aggregator that wires RestartTracker + Ports + Pipelines + per-stream FileLoggerhealthServer.ts—GET /healthreturning{status, streams}(no Express, just Nodehttp)FileLogger— size-based per-stream stderr rotationstreamSpecsLoader.ts— pulls(obs_source_name, url)rows from the webui's SQLiteindex.ts— entry point; reads env, opens DB via existinggetDatabase(), wires SIGINT/SIGTERMREADME.md— NSSM Windows install procedure, env-var reference,/healthexamplePhase 1.3 — convertBrowserToMedia
scripts/convertBrowserToMedia.ts(CLI) +scripts/convertBrowserToMedia/:transform.ts— pure data transform; usesbuildFfmpegSourceSettings(single source of truth from 1.1)obsRunningCheck.ts—taskliston Windows /pgrepon macOS; injectable execbackup.ts— timestampedscenes.backup.<ISO>/with round-trippable restorerunConversion.ts— orchestrator: refuses to run if OBS is open; refuses if mapping is empty;--dry-runwrites.diff.json; non-dry runs always backup firstREADME.md— usage, safety contract, mapping sources (--supervisor-urlor--mapping-file), rollbackHousekeeping (orthogonal pre-Phase-1 fix)
app/api/__tests__/{teams,streams}.test.ts— pre-existing broken stubs (GET import commented out, assertions referencing un-invoked mocks). Wired the handler calls in and tightened the mocks. Also syncedpackage-lock.jsonwith the optionalDependencies move from commit7597699.Numbers
npm test,npm run type-check,npm run audit:sqlite-opensall clean.Test Plan
npm test— all 137 unit tests greennpm run type-checknpm run audit:sqlite-openscreateStreamGroupV2(..., useFfmpegSource: true)against the production OBS host via WebSocket — deferred to Phase 4.1 dress rehearsal per plan macOS preface/healthreports all streamsrunning, verify a killed streamlink respawns within ≤1s, verify 3-in-30s escalation flips status toescalated— deferred to dress rehearsalSaT.jsonwhile OBS is closed; review.diff.json; commit run; reopen OBS and assert all 7 source-switcher inputs resolve and 4-Screen layout renders — closes G1.3setActivein 60s + 7 concurrent Streamlinks for 30 min) — not in this PRRollback
useFfmpegSource: false(V2 behaves as V1).createStreamGroup— V2 is additive.restoreFromBackup({ backupPath, sceneFile })orcp scenes.backup.<ISO>/SaT.json /path/to/SaT.json.Notes
streamstable via the existinggetDatabase()singleton — no newsqlite.open()site, no allowlist change.V2 runs alongside V1 (lib/obsClient.js createStreamGroup is untouched as the rollback target). The per-input choreography is factored into lib/streamInputConfig.js so it can be unit-tested without pulling the obs-websocket-js ESM module through Jest's transformer: - buildStreamInputConfig({ url, useFfmpegSource }) pure: returns { inputKind, inputSettings } for browser_source (V1 parity) or ffmpeg_source (V2 new path). - createStreamInput({ call }, { sceneName, inputName, url, useFfmpegSource }) OBS choreography: CreateInput → SetInputSettings(overlay:false) → SetInputMute (browser_source only — ffmpeg_source audio routes natively). Takes an injected `call` so tests can verify the payloads without an OBS connection. createStreamGroupV2 mirrors V1's scene/text/color/position/bounds work but delegates the input creation to createStreamInput. Defaults to useFfmpegSource:false so callers that don't set the flag behave identically to V1. Per-call rollback is `useFfmpegSource:false`; full-system rollback is `createStreamGroup` (V1). restart_on_activate stays false for ffmpeg_source per the plan — Phase 2.4 will flip it after the supervisor lands. Acceptance per plan: - both paths green in unit tests: 7 new tests in lib/__tests__/obsClient.test.js (4 helper, 3 choreography) - integration test exercises mixed mode: [PROD-HOST] tagged, deferred to Phase 4.1 dress rehearsal per the plan's macOS preface - rollback: V1 untouched + V2 default flag = V1 behavior Full suite: 63 passing (was 56; 8 of the original 48 were pre-existing broken stubs fixed in the previous commit, +7 new this commit).First piece of the Streamlink supervisor. Pure logic, no I/O, so it can be unit-tested without spawning real processes: - record(streamId, now): logs a restart timestamp - shouldEscalate(streamId, now): true once `max` restarts have occurred within `windowMs` - forget(streamId): clears history (e.g., on operator manual reset) Defaults are 3 restarts in 30s per the plan; windowMs and max are configurable for tests and future tuning. 8 new tests cover the policy edges (single restart, restarts that slip past the window, per-stream isolation, custom thresholds). Spawn mechanics, /health endpoint, and stderr rotation land next.Process-supervision primitives behind the Streamlink supervisor: - PortAllocator: hands out sequential UDP ports from a base, reuses the lowest released slot first, throws PortExhaustedError when the budget is full. Used to assign each stream its own ffmpeg → OBS UDP target (9001, 9002, …). - buildStreamlinkCmd / buildFfmpegRelayCmd (commands.ts): pure builders for the per-stream CLI pair. Streamlink writes raw TS to stdout with --twitch-disable-ads + --hls-live-restart; ffmpeg -c copy -f mpegts pushes to udp://127.0.0.1:<port>?pkt_size=1316. Each builder accepts a custom binary path for the Windows install layout. The ffmpeg builder also returns obsInputUrl ("udp://127.0.0.1:<port>") for direct injection into OBS ffmpeg_source.input. - StreamPipeline: owns one streamlink+ffmpeg pair. Injectable spawn fn so the orchestration is fully unit-testable without real subprocesses. Pipes streamlink.stdout → ffmpeg.stdin, cascades a kill on the sibling when either child exits, and invokes onExit/onStderr exactly once with the originating source tagged. Status: pending → running → exited. Delivery shape locked per design alignment: UDP MPEG-TS on the OBS host (lowest-moving-parts option; localhost loss ≈ 0). Supervisor host is the same Windows box as OBS (NSSM-wrapped, to come). Suite: 90 passing (was 71; +19 across 3 new test files). Still ahead in 1.2: the Supervisor aggregator that wires RestartTracker into the pipeline lifecycle, the /health HTTP endpoint, per-stream stderr log rotation, the entry script, and the NSSM service config.Supervisor ties RestartTracker + PortAllocator + StreamPipeline into a working per-host service: - start(spec): allocates a UDP port, constructs a pipeline, wires onExit to record the restart and either respawn (via injected enqueue → setImmediate by default, so the new pipeline is built on a clean tick) or flip the stream to status='escalated' once the tracker hits 3 restarts in 30 s. - stop(streamId)/stopAll(): tears down pipelines, releases ports, clears tracker history. - get(streamId)/list(): serializable per-stream state snapshots (streamId, port, status, restartCount, obsInputUrl, lastExit*). healthServer.handleHealthRequest serves GET /health from the SnapshotProvider interface (Supervisor satisfies it). Returns { status: 'ok' | 'degraded', streams }: 'ok' when every stream is running, 'degraded' when at least one is exited/escalated. Empty stream list is 'ok'. Non-/health → 404. Non-GET → 405. The startHealthServer helper wraps it with the built-in `http` module (no Express dep). Both modules are fully unit-tested (6 + 5 tests), all injection-based so no real subprocess or socket is opened by the tests. Suite is now 101/13 (was 90/11). Type-check clean. Still ahead in 1.2: per-stream stderr log rotation, an entry script that reads stream specs and wires Supervisor + healthServer + SIGTERM handlers, and the NSSM service config for Windows installation.Closes the remaining pieces of the Streamlink supervisor: - FileLogger: per-stream stderr capture with size-based rotation (active.log → .1.log → .2.log …). Cascades up to `retain` files and drops the oldest past that, so disk use is bounded by `maxBytes × (retain + 1)`. - loadStreamSpecs (streamSpecsLoader): reads (obs_source_name, url) from the webui's `streams` table, filters out rows with empty values, and returns StreamSpec[] for the Supervisor. Rejects table names that aren't safe SQL identifiers (defensive against env-var injection — SQLite identifiers can't be parameterized). - startRuntime: wires Supervisor + RestartTracker + PortAllocator + per-stream FileLogger + /health server, loads initial specs from the DB, returns an idempotent shutdown() that stops every pipeline, closes the loggers, and shuts the HTTP server. Fully unit-tested (3 tests) with injected db + spawn. - index.ts: production entry point. Reads env vars (with sensible defaults documented in README), opens the webui DB via the existing getDatabase() singleton (no new sqlite.open() site to allowlist), wires SIGINT/SIGTERM to runtime.shutdown(), and delegates to startRuntime. Pure imperative wiring, untested by design — every component it composes is covered. - README: NSSM install procedure for the Windows OBS host, env-var reference, /health example, operating notes (add/remove/escalate a stream), and explicit non-goals (no non-loopback bind, no auth, no cross-host failover, no plugin DB integration — R4 stays post-event per the iter-3.4 plan). - package.json: `npm run supervisor` script. - healthServer test: tightened the fake Response helper so it type-checks cleanly under strict TS (no behavior change). Suite: 113/16 passing (was 101/13; +12 across fileLogger, loader, runtime). Type-check + sqlite-opens audit both clean. Task 1.2 done at the unit-test layer per plan acceptance: - S2 integration semantics: covered by RestartTracker + Supervisor respawn tests - stderr rotation caps disk: FileLogger retention math + diskBytes - Windows respawn primitive selected and unit-tested: NSSM (docs + AppRestartDelay/AppThrottle config in README) [PROD-HOST] integration (real streamlink + ffmpeg + OBS) is the Phase 4.1 dress rehearsal per the plan; not in scope for the TDD-test layer.Rewrites browser_source inputs in an OBS scene-collection JSON into ffmpeg_source inputs pointing at the Streamlink supervisor's per-stream UDP endpoints, with the safety contract called out by Phase 1.3 G1.3. Decomposed into small, individually tested modules: - transform.ts: pure data transform on the parsed scene JSON. Replaces id/versioned_id and settings on browser_source inputs present in the mapping (sourceName → udp://127.0.0.1:port), passes through every other source kind unchanged, returns a diff structure that records before/after for auditing. Uses buildFfmpegSourceSettings from lib/streamInputConfig — single source of truth for the ffmpeg_source settings shape across V2 createStreamGroup (Phase 1.1) and this script. - obsRunningCheck.ts: cross-platform "is OBS running" probe. Windows uses tasklist filtered on IMAGENAME=obs64.exe; macOS uses `pgrep -x OBS`. Exec is injected for unit testing. - backup.ts: timestamped copy to <backupRoot>/scenes.backup.<ISO>/ with a round-trippable restore. Refuses to overwrite an existing backup dir — operator must pick a fresh timestamp or clean up previous backups before rerunning. - runConversion.ts: orchestrator. Refuses to run if OBS is detected (skipped under --dry-run). Refuses to run if the mapping is empty. Backs up first, then writes the converted JSON. --dry-run emits a sibling <scene>.diff.json instead of mutating the file. - convertBrowserToMedia.ts: CLI entry. Mapping comes from either --supervisor-url <url> (fetches /health, builds streamId→udp map from the live supervisor) or --mapping-file <path.json> (static file for testing / disaster scenarios). Pure imperative wiring, untested by design. - README: usage, safety contract, mapping sources, rollback procedure, and explicit non-goals (no scripted OBS-launch validation — deferred to operator dress rehearsal per plan G1.3; no source-switcher rewrites — Phase 0.7.3 coverage check still owns that surface). Suite: 137/20 passing (was 113/16; +24 across transform, OBS-running probe, backup, runConversion). Type-check + sqlite-opens audit clean. Task 1.3 closed at the unit-test layer. [PROD-HOST] round-trip (close OBS → run script → reopen OBS → validate all 7 inputs + 4-Screen layout) is the operator step that closes G1.3 per the plan; this commit gives them a reversible, dry-run-able script to do it. Remaining in Phase 1: Task 1.4 load test (100 setActives in 60s + 7 concurrent Streamlinks for 30 min), which is mostly operator drive against the production host once Phase 0.5 prerequisites are met.