This page shows how the machine works and which parts are switched on right now, based on the environment this service was started with. Use the toggle in the corner to switch between this view and the engineering view.
FastAPI service hosting the news bounded context: three cron-driven stages
(ingest → extraction → generation) coordinated through timestamp columns, plus synchronous
on-demand endpoints. Statuses below are derived from NewsSettings
(env prefix NEWS_*) — adapter selection is "real when configured, else in-memory fake".
Secret values never reach this page; only presence booleans do.
The whole system as a wired node graph — follow the white wires left to right. Diamond-headed gray nodes are decisions with True / False exits. Drag the board to pan, scroll to zoom, drag nodes to rearrange.
Execution wires (white) = control flow per tick / request. Colored wires = data dependencies; green source nodes are live adapters, amber ones are the in-memory fakes selected by the current env. Same sanitized snapshot as the document view.
Four stations on a conveyor belt. Each one finishes its work, stamps it, and the next one picks it up. If one station has a bad day, the others keep running — work waits, nothing is lost.
Drivers in application/schedulers.py, composed in app/main.py lifespan.
Every tick: Postgres advisory lock → read run-state baton → stage compute → stamp.
A failing tick logs and waits for the next minute; pending runs retry.
Between … and …, at most every … minutes, we pull the top … stories from 7 Google News categories, skip ones we already have, fetch the full text of each new story, and save everything.
ingest_tick: cheap gate (switch + window) → googleNewsIngest_lock →
interval gate → per category: search_for_news → flatten (cap
) → SHA-256 URL dedup → image HEAD validation →
body fetch per new item → persist_batch in one tx + position reconciliation.
An AI reads every new story and answers: what is this actually about? Stories are grouped into ongoing storylines ("topics"), reusing ones we already track, and link-roundup pages are flagged so they don't pollute briefings.
extraction_tick: newsTopicExtraction_lock → pending runs? →
drain the global backlog in batches of
through TopicExtractorAgent (structured JSON via output_schema) →
per-article decisions (existing / related / new topic) → persist topics +
news_sources_topics links in one tx → stamp topics_extracted_at.
For each user we pick the strongest fresh storyline they care about, then an AI writes a short executive briefing: a new article, or an update to the story so far.
generation_tick: newsArticleGeneration_lock → pending runs? →
PERSONALIZED pass: stream DS profiles for active users → matcher LLM + lexicographic
Google-position re-rank →
freshness gate (newest source must beat the prior by
min) →
ArticleWriterAgent writes/updates → persist + citations → stamp
articles_generated_at.
Finished briefings are stored, and nothing is pushed anywhere. Clients get news by asking for it: every interaction goes through the HTTP API, which returns articles in the same call.
The pipeline ends at persistence: generation_tick stamps
articles_generated_at and returns. There is no push transport, no
delivery records, no fan-out. Read paths are the synchronous endpoints
(/generate, /refresh, follow-up questions);
scheduled PERSONALIZED rows persist in generated_articles
for the pull-based read endpoints this rework introduces next.
Stages never call each other. Each one leaves a timestamp in the database, and the next stage looks for work missing its own stamp. No message broker to run, monitor, or break — and if the service restarts mid-way, it simply picks up where the stamps left off.
google_news_ingest_runs: completed_at →
topics_extracted_at → articles_generated_at.
Per-stage Postgres advisory locks (lockAtMostFor 60/30/30 min) make multi-node
deployments single-runner per stage. Repos in
infrastructure/db/repositories/ingest_run_repo.py.
Story collection isn't constant — it runs inside a daily window, with a minimum pause between runs. Everything downstream wakes up every minute and only works when there is something new.
_cheap_ingest_gate (master switch + clock window, evaluated before the lock)
then _interval_gate_passes (max(completed_at) spacing, evaluated
inside the lock so racing nodes can't double-ingest).
Green is when new story collection is allowed. Outside the window the collector stays quiet; understanding and writing still process whatever is already waiting.
Ingest cron fires every minute and self-gates to ~1 run/min inside the window. Extraction & generation ticks have no clock gate — only the pending-run check.
Four ways value reaches a user — one scheduled, three on demand.
One cron path + three HTTP entry points. Routers are thin; each consumes exactly one
service-level provider from app/news/api/deps.py.
ingest_tick → run_ingest (7 categories, dedup, bodies).extraction_tick → run_extraction → topic decisions.generation_tick → PERSONALIZED pass → matcher + freshness gate.ArticleWriterAgent → persist + citations.MatchingService.match_topic_ids: matcher LLM over-fetches , lex Google-position re-rank.OnDemandArticleService.generate: DS profiles → one CURRENT_PROFILE block (multi-reader block instructs SHARED-interest calibration); unavailable profiles drop out, never block.USER_REQUEST (user_id only for a single reader; group → null) → 200 · list[ArticleResponse].ingest_scoped_query: free-text q search → flatten → SHA-256 dedup → bodies → insert_new_articles (no Google-position table).run_extraction(only_ids=…) — scoped to what was ingested, not the global backlog.ScopedBriefingService: freshness gate over the (topic, USER_REQUEST, user_id) chain → update mode when a prior exists; SKIP returns the current head.USER_REQUEST with parent_article_id → 200 · list[ArticleResponse] (ADR 0006).FollowUpAgent — exactly non-blank questions or []; retries.200 with empty list. ?testDelay = sim mode (3 canned, no LLM).Every outside system has a stand-in built in. Green means the real connection is configured; amber means the safe built-in stand-in is doing the job — useful for development and testing, invisible to the code around it.
Boundaries are typing.Protocol ports (application/ports.py) with in-memory
fakes. Selection happens once, in composition.py / api/deps.py:
real adapter when its env config is present, else the fake — no code change between environments.
The shared database — the pipeline's memory and its conveyor belt.
SQLAlchemy 2.0 async + asyncpg; Alembic migrations. DSN composed from parts when a host is set (ECS), else one URL.
The AI that reads stories, picks storylines, and writes briefings — four different jobs, one model.
ADK-native Gemini via AdkJsonRunner (sole google.adk import). Model ; unset key falls back to GEMINI_API_KEY / GOOGLE_API_KEY.
Finds what Google News is showing right now, per category.
search_for_news — section search by topic_token; locale hardwired us/en.
Fetches the full text of each story so the AI reads articles, not headlines.
fetch_page_content — body scrape, tight timeouts; both keys required for the live provider.
Knows each user's interests — the "what does Maria care about?" service.
ProfileService port — profile by user + paged profile stream for the PERSONALIZED pass.
Who is active — so briefings are only written for real, current users.
UserDirectory — read-only over the shared icebreaker users table when a DB host is configured.
Five kinds of briefing exist in the schema. Two are live; three are not currently produced.
GeneratedArticleKind — all five stay in the enum + DB CHECK. The non-produced
kinds are re-enableable; do not remove their values.
The scheduled briefing, matched to each user's interests.
The only kind the scheduled generation entry point produces.
Written on the spot when a question comes in — for everyone, for one reader, or for what several readers share.
On-demand via POST /generate; user_id set only for single-reader rows.
The old profile-driven briefing — superseded by personalized questions above.
Enum + CHECK kept; the united /generate endpoint persists USER_REQUEST for both branches (ADR 0005).
A "biggest stories of the day" digest — built, currently off.
Coded + schema-supported; live entry point never produces it.
A per-category digest (tech, sports, …) — built, currently off.
Same status: enum + CHECK present, generation pass disabled.
Think of it as rings: the rulebook sits in the middle and knows nothing about databases, AI vendors, or HTTP. Each outer ring may use the rings inside it — never the other way around. That's why a stand-in can replace any outside system without touching the rules.
Layered DDD, one bounded context (news). Inward-only imports; externals are
Protocol ports implemented in infrastructure/external/ with in-memory fakes.
Wiring via FastAPI Depends (api/deps.py) and the composition root
(composition.py). Every numeric threshold lives in constants.py
with rationale.
Full context: app/news/CONTEXT.md · decisions: docs/adr/.