Architecture

ARCHITECTURE.md

KalshiTrader Architecture

The system map. For history and decision context see PLAN.md; for the regulatory posture see analysis_l.md.

Data flow

Kalshi REST ──┬─ FastDiscovery (20s, min_open_ts) ─┐
              ├─ MarketSync (2m open /10m unopened)├─ Markets.sync_markets ─ diff → market_events + "markets:new"
              └─ MarketQuotes (per-series, on tick)┘        │
                                                   write-through marks + throttled market_snapshots
Kalshi WS ────┬─ trades ─ TradeBuffer ─ batched → trades table + "market:trade:T"
              ├─ orderbook ─ OrderbookTracker (books for maker fills)
              └─ fills/positions → Trading (user account state)
Gemini WS ────── GeminiSocket → GeminiQuotes ETS (bookTicker + depth5, move tracking)
Polymarket WS ── PolymarketSocket → PolyQuotes ETS (CLOB books for linked markets)
Venue REST ───── VenueSync (Polymarket gamma + Gemini events) → markets (PM:/GM: tickers)
Coinbase/Pyth ── CandleSync/Backfill → candles (1h × 18 symbols, 15m liquid five;
                 Coinbase-first behind a circuit breaker since the Pyth API lockout)
NOAA/ESPN ────── WeatherClient / SportsData; EconCalendar (FOMC/NFP/claims/CPI)

Cross-venue: Venues.Equivalence links same-event markets (deterministic ticker parsing for crypto ladders, fuzzy titles elsewhere) → market_links

  • venue_spreads; Workers.ArbPaper locks crossed books on paper with

venue-true fees, records leg quote ages/depth, and is gated by Research.ArbEvidence family verdicts (a settlement divergence = NO-GO). Workers.VenueSettlement fetches real Gemini/Polymarket resolutions.

ARM-NOW alerting (issue #205)

ArbPaper fires a CRITICAL "ARM AND EXECUTE" alert on :ops whenever all of the following are true during a paper-arb tick:

  • config :kalshi_trader, :arb_executor_live_enabled is true
  • ArbExecutor.flight_check/0 returns zero :fail items (:warn items are allowed — operators know to review but they don't block)
  • The executor is not already armed for the current ISO week
  • The spread clears the same gates as the #136 "EXECUTABLE ARB" alert: net ≥ :arb_alert_net_cents (default 4¢), depth ≥ :arb_alert_min_size_cents (default $5), and link is confident
  • The current UTC hour is outside quiet hours (config :kalshi_trader, :arb_arm_now_quiet_hours, {22, 6} — suppresses 22:00–05:59 UTC; set to nil to disable)

A per-link 30-minute cooldown prevents repeat spam on sticky spreads (independent of the existing EXECUTABLE ARB 30-minute suppress).

The alert is intentionally distinct in wording from "EXECUTABLE ARB" (#136) so operators can tell them apart at a glance. It includes tickers, net edge, depth, and a deep link to /arb (built from KalshiTraderWeb.Endpoint.url/0, which resolves to the full production URL via PHX_HOST, falling back to the relative /arb path if the endpoint is unavailable).

Scope: quiet hours apply only to ARM-NOW alerts. Other critical ops alerts (missed legs, auto-pause, auto-halt) are unaffected.

Strategies consume PubSub topics + periodic {:tick, now}; signals persist and broadcast; the execution engine turns them into orders.

Layers and key modules

LayerModules
IngestionIngestion.{FastDiscovery, SocketManager, TradeBuffer, OrderbookTracker}, Workers.{MarketSync, CandleSync, CandleBackfill, SeriesSync, SettlementSync}
Domain contextsMarkets (sync/diff/lifecycle), MarketData (trades/candles), Trading (signals/orders/fills/performance/audit), Weather, Exposure, Crypto (asset registry), Models, ApiTokens
StrategiesStrategies.Strategy behaviour + Runner + Strategies (seeding/supervision); implementations in Strategies.*; statistics in Strategies.{CandleStats, MarketQuotes}, ML.Logistic
ExecutionExecution.{Engine, RestingOrders, PositionManager, RiskManager, FillStats}, Fees
VenuesVenues.{Venue, Polymarket, Gemini, Equivalence, Meta, Fees, FeeVerification}, Ingestion.{GeminiSocket, GeminiQuotes, PolymarketSocket, PolyQuotes}, Workers.{VenueSync, ArbPaper, VenueSettlement, BookSnapshot}FeeVerification extracts actual fee fields from order payloads and compares against Venues.Fees (issue #183; see VENUES.md § "Fee model & verification")
ResearchBacktest (PIT replay + exit sim + maker blending), Research.{ArbEvidence, SpreadStudy, VenueLeadLag, VenueCalibration, BookStudy, FillRealism, Correlation, AllocationAdvisor, MonteCarlo, Stress, EventStudy}, mix tasks kalshi.{backtest,sweep,event_study,seasonality,archive,archive.verify,token,webhook,export,fees.verify}
Reporting/opsWorkers.{MorningReport, WeeklyReport, ResearchReport, PilotDigest, AutoTune, SeasonalityTune, Lifecycle, Watchdog, DataQuality, FillQuality, WeatherCalibration, ModelFit, EquitySnapshot, LiveRecon, WatchlistAlerts, Backup, RestoreCheck, Prune, LiveOrderSync}, Alerts.{Notifier, Push} (Discord + web push, per-device all/critical filters), Webhooks, ErrorLog (in-memory ring + persisted errors table), Reports
WebSix-group nav (Tracker · Markets · Crypto · Trading · Venues · Portfolio); LiveViews incl. /positions, /event/:ticker, /search, /reports, public /performance, /report, /developers, /terms, /disclaimer; PWA (manifest + service worker + push); ApiController (/api/signals, /api/kill, /api/health)

The strategy contract

init(params) → {:ok, state}; subscriptions(state) → [topic]; handle_event(event, state) → {:signals, [map], state} | {:noop, state}. Events are PubSub messages plus {:tick, DateTime} every tick_ms. Signal maps: market_ticker (required), side, action, price_cents (the ask), bid_cents (enables maker posting), count (pre-Kelly), confidence (0..1 — feeds Kelly sizing and calibration), rationale.

Strategies must be point-in-time safe: inside backtests the process key :backtest_as_of is set; MarketQuotes then serves historical snapshots, CandleStats truncates history, and anything touching live feeds must check the key (see MomentumFifteen.live_spot).

Execution pipeline

signal → Kelly sizing (confidence × virtual bankroll, quarter-Kelly, opt-out via sizing: "fixed") → RiskManager.check (kill switch; order cost; per-market, per-underlying, total exposure caps; mode-specific limits with live micro-caps) → entry:

  • paper maker (default): rest at bid_cents, RestingOrders fills off the trade feed with queue-position modeling (size-ahead from the tracked book), taker fallback at maker_timeout_min;
  • paper taker (entry: "taker"): immediate fill at ask + slippage + fee;
  • live: requires live_approved; resting orders reconciled by LiveOrderSync.

Exits: PositionManager (TP/SL/settle-freeze, per-strategy exits params). Trading-hours filters (trading_hours/exclude_hours params) gate ticks in the live Runner AND in backtests. Lifecycle circuit-breaks on trailing-7d drawdown, auto-culls deep losers, auto-demotes live strategies on realized loss, and announces first-time live-gate passers (all audited). Live extras: LiveGate (readiness + one-click pilot at $0.25/order), live maker entries with timeout cancels, LiveOps.cancel_all!, nightly LiveRecon balance reconciliation, intraday drawdown + unexplained balance-move alerts.

Money conventions

All prices/PnL in integer cents; strikes in float dollars. Fees: Fees.taker_fee_cents = ⌈0.07·C·P·(1−P)⌉ in exact integer math; maker = 0. Position accounting nets buys/sells per (strategy, ticker, side); settled positions pay 100¢ × remaining contracts on a win. Invariant tests: test/kalshi_trader/money_invariants_test.exs.

Audit rules

Every strategy-config mutation goes through Trading.upsert_strategy_config(attrs, actor) and writes audit_log (before → after, actor user:<email> | system:<source>). Automated actors: system:{seed, auto_tune, seasonality, circuit_breaker, lifecycle, lifecycle_cull, live_demotion, live_gate, arb_evidence}. AutoTune applies changes only walk-forward-validated, opt-in, one grid step/night. Statistical inputs (candle stats, fitted models, measured weather sigma, maker rate) refresh continuously and are not audited — they're data, not configuration.

Worker schedule (Oban cron; watchdog auto-verifies coverage)

See the crontab in config/config.exs — it is the source of truth. Rough map: market sync 2m/10m, candles 2m, watchdog 5m, live-order sync 5m, settlement + series hourly, equity snapshots hourly, model refit + autotune nightly, morning report + lifecycle daily, weather calibration daily, fill quality + weekly report Mondays, backup nightly (+ offsite copy), restore check monthly, prune daily.

Gotchas (hard-won)

  • 2026 Kalshi API: prices arrive as <base>_dollars strings and counts as <base>_fp; status query param unopened ↔ object status initialized; markets universe ≫ the 60k sync cap (hence FastDiscovery + quote-driven snapshots).
  • Novelty: every ladder window is a "new event"; data.new_series is the news signal, new_event is diluted (millions/week).
  • Timestamps in Postgres are naive UTC; raw SQL params must be cast ::timestamp with naive values, never ::timestamptz.
  • config/*.exs changes (including git branch switches touching them) wedge the dev code reloader — restart the server.
  • Research CLI tasks set KALSHI_MINIMAL=1 (Repo+PubSub only boot); never run a full second node against the same DB.
  • Kill processes by the port-4001 owner pid, never by image name (erl.exe matching also catches unrelated processes).
  • Backups: nightly dump + OneDrive offsite copy + monthly restore proof; the local archive warehouse (mix kalshi.archive, nightly Task Scheduler job) is the cloud-loss recovery path — kalshi.archive.verify alerts if it lags (a restore drill caught it silently stale once; see docs/RESTORE_DRILL.md).
  • Work tracking: GitHub issues labeled roadmap, closed with evidence comments (commit, tests, measured results) + evidence-closed label.

Settlement and oracle divergence coverage

There is no OracleMonitor module in this codebase — the claim in an earlier planning digest (managed\_next20 item) that an orphan stub existed was incorrect. No such module was ever committed or removed; there is nothing to replace or delete.

Proactive settlement/oracle risk is covered by three layers:

LayerModule(s)What it catches
Rules drift (pre-settlement)Research.TermsSentinel (cron daily 06:30)Venue edits settlement-terms text after a link is graded — source, measurement time, inclusivity, early-settlement clause changes. CRITICAL alert on confident (traded) links.
Post-settlement truenessWorkers.DataQualityResearch.ArbTruth + Research.DivergenceForensicsArb settlement trueness audit; divergence forensics on resolved markets.
Resolution syncWorkers.VenueSettlement + Workers.SettlementSyncFetches real Gemini/Polymarket resolutions; blocks execution on divergence verdicts.

If a future live oracle-feed monitor is wanted (e.g. real-time oracle price vs expected settlement price during the settlement window), that would be a brand-new feature added as its own worker/module — not a replacement for any stub.

Live arb analysis pack (#207)

Research.LiveArbAnalysisPack and mix kalshi.arb.analysis_pack are the #207 evidence path — a read-only post-trade analysis tool that fires on the first real two-leg executions. The pack is NOT a trading surface; it does NOT arm, enable live orders, or place any orders. Calling it or the mix task can never cause capital to move.

When arb_executions has no mode = "live" rows the pack returns a labelled template stub (:no_live_trades). When rows exist the pack covers:

  • Realized vs quoted edge per execution
  • Fees actual vs Venues.Fees model (noting if FeeVerification #183 is present on the branch)
  • Latency at each hop (from audit_log entity arb_executor)
  • Slip (far fill vs quote; Kalshi fill assumed at quote per executor design)
  • GeminiRecon reconciliation status at report time
  • Status breakdown (filled_both / stub / unfilled / failed)

Do not confuse this module with any live-enabling path. The arm/1 flow, live_gates, and arb_executor_live_enabled config are unrelated to this report.