jarvis/docs/backend/BACKEND_HANDOFF.md

Jarvis Backend — Build & Deploy Handoff (for Codex on Proxmox)

You are building the server side of Jarvis: a self-hosted RSS news-correlation platform. A finished SwiftUI iOS client already exists and will connect to you unmodified — so the API shapes below are not suggestions, they are a contract. The canonical spec is API/contract.md in this repo; this document adds the implementation plan, the algorithms, the client-compat gotchas, and the Proxmox deployment.

Stack (already decided): Python 3.11+ · FastAPI · Uvicorn · SQLite (via SQLModel) · scikit-learn TF-IDF for correlation · rule-based scoring. No external API keys. No auth (local network / self-hosted only).

---

0. TL;DR of what to build

A single FastAPI app that does four jobs:

1. Ingest — poll RSS feeds on a schedule, parse articles, track feed health.
2. Correlate — cluster articles from different sources into Stories (TF-IDF + cosine).
3. Score — compute a server-side signalScore (5 components) + topic + consensus/conflict.
4. Serve — the REST API under /api/v1 and a WebSocket at /ws, broadcasting live events.

Then deploy it as a systemd service inside a Proxmox LXC container, bound to 0.0.0.0:8080, reachable from the iPhone on the LAN.

---

1. NON-NEGOTIABLE client-compatibility rules

The iOS client decodes JSON with keyDecodingStrategy = .convertFromSnakeCase and dateDecodingStrategy = .iso8601, and it silently drops any object that fails to decode. So:

1. Dates must be exactly YYYY-MM-DDTHH:MM:SSZ — UTC, the literal Z, and no fractional seconds. Pydantic's default (...+00:00 with microseconds) will FAIL to decode and stories will vanish. Force the format with a serializer (see §4).
2. JSON keys: camelCase (signalScore, sourceCount, pollIntervalSeconds, publishedAt, nextCursor, hasMore, …). FastAPI: set field aliases via alias_generator=to_camel, serialize with response_model_by_alias=True (default).
3. Exact query-param names the client sends to GET /stories: limit, after (the cursor), topic, min_signal. Read those names verbatim.
4. DELETE /feeds/:id must return HTTP 204 (no body). The client checks for 204.
5. POST /feeds returns 201 with the created feed object. The client sends {"url": ..., "name": ..., "pollIntervalSeconds": ...} — accept camelCase.
6. Base URL is http://<host>/api/v1 and WebSocket is ws://<host>/ws — the client builds these strings literally, so REST lives under /api/v1 and the socket at the root path /ws. Plain http/ws (not TLS) is expected.
7. No CORS needed (native app, not a browser), but harmless to enable *.
8. Error responses use {"error": {"code", "message", "status"}} (see contract §Error).

---

2. Endpoints (all under /api/v1 unless noted)

Full example payloads are in API/contract.md. Summary:

Method	Path	Returns	Notes
GET	/health	{status, version, storiesCount, feedsCount, uptime}	client calls on launch to validate connection
GET	/stories	{data[], nextCursor, hasMore, total}	sorted by signalScore desc; params limit(≤100, def 20), after, topic, min_signal
GET	/stories/{id}	StoryDetail (adds timeline[])	404 story_not_found if missing
GET	/articles/{id}	Article (full body for offline cache)	404 if missing
GET	/feeds	{data: [Feed]}	all feeds + health
POST	/feeds	Feed (201)	body {url, name, pollIntervalSeconds?}; 400 invalid_url if not a real feed
DELETE	/feeds/{id}	204	404 feed_not_found if missing
WS	/ws (root, not /api/v1)	event stream	see §6

Object shapes (camelCase, dates as ...Z)

• StorySummary: id, headline, summary, topic, signalScore, scoreBreakdown, sourceCount, sources[], consensus, conflict, updatedAt, createdAt
• scoreBreakdown: {sourceAuthority, freshness, localRelevance, crossSourceConfirmation, topicImportance} (ints, sum == signalScore)
• StorySource: {id, name, url, publishedAt, isBreaking}
• StoryDetail: StorySummary fields minus sources, plus timeline[]
• TimelineEntry: {articleId, source, headline, publishedAt, isBreaking}
• Article: {id, storyId, source, sourceUrl, headline, body, imageUrl, author, publishedAt}
• Feed: {id, name, url, health, pollIntervalSeconds, failureCount, lastFetchedAt, articleCountToday}
  • health ∈ {"active","failing","dead"}
• topic slugs used by the app's filter pills: finance, tech, politics, africa (classify everything into one of these; fall back to the best match).

---

3. Suggested project layout

server/
  app/
    main.py          # FastAPI app, lifespan: init DB, seed, start poller
    config.py        # tunables: thresholds, lexicons, source-authority map
    db.py            # SQLModel engine/session, init_db()
    models.py        # Feed, Article, Story (SQLModel tables)
    schemas.py       # Pydantic response models (camelCase aliases + Z dates)
    ingest.py        # RSS poll + parse + feed-health state machine
    correlate.py     # TF-IDF clustering: assign articles -> stories
    scoring.py       # signal-score components, topic classify, consensus/conflict
    events.py        # WebSocket ConnectionManager + broadcast + 30s ping loop
    scheduler.py     # asyncio loop: poll due feeds, correlate, score, sweep stale
    routers/
      stories.py  articles.py  feeds.py  health.py  ws.py
  seed_feeds.py      # default feed list inserted on first run
  requirements.txt   # already in this folder
  jarvis.service     # systemd unit (see §8)
  README.md

---

4. Data model (SQLite via SQLModel)

• Feed: id (pk, "feed_"+short uuid), name, url, health, poll_interval_seconds, failure_count, last_fetched_at (nullable), created_at. articleCountToday is computed at query time (count of this feed's articles with created_at >= today 00:00 UTC).
• Article: id ("art_"+uuid), feed_id (fk), story_id (fk, nullable), source (feed name), source_url, headline, body, image_url (nullable), author (nullable), published_at, guid (unique — dedup key), created_at, match_text (lowercased headline + body, for TF-IDF).
• Story: id ("story_"+uuid), headline, summary, topic, signal_score, the 5 breakdown ints, consensus (nullable), conflict (nullable), source_count, is_stale (bool), created_at, updated_at.

Store all datetimes as naive UTC. Serialize with this exact helper so the client's .iso8601 accepts them:

from datetime import datetime, timezone
from pydantic import field_serializer

def iso_z(dt: datetime) -> str:
    if dt.tzinfo is not None:
        dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
    return dt.strftime("%Y-%m-%dT%H:%M:%SZ")   # NO microseconds, literal Z

Apply via @field_serializer("publishedAt","updatedAt","createdAt","lastFetchedAt") returning iso_z(value) (return None for null lastFetchedAt).

Camel aliases:

from pydantic import BaseModel, ConfigDict
from pydantic.alias_generators import to_camel

class Schema(BaseModel):
    model_config = ConfigDict(alias_generator=to_camel, populate_by_name=True)

---

5. The algorithms

5a. Ingest (ingest.py)

• Fetch each due feed with httpx.AsyncClient (timeout ~10s), parse bytes with feedparser (run in asyncio.to_thread, it's blocking).
• For each entry: dedup by guid (entry.id or link). Extract headline (title), body (richest of content[].value / summary, HTML-stripped with BeautifulSoup), image_url (media:content / enclosure / first <img>), author, published_at (entry.published_parsed → UTC; fall back to now).
• Feed-health state machine on each poll:
  • success → health="active", failure_count=0, last_fetched_at=now.
  • failure → failure_count += 1; failing at ≥1, dead at ≥5 (tunable).
  • On any health change, broadcast a feed.health WS event.

5b. Correlate (correlate.py)

Goal: same real-world event across sources → one Story; stable Story ids.

• Consider only articles from a rolling window (e.g. last 72h).
• Single TfidfVectorizer fit per cycle over (existing recent stories' aggregate text + the new unclustered articles' match_text). English stopwords, ngram_range=(1,2).
• Greedy online assignment per new article (process oldest→newest):
  • cosine-compare against current cluster representatives;
  • if max_sim >= 0.22 (tunable) and within the time window → join that story;
  • else → create a new Story seeded by this article (append its vector as a new rep).
• After assignment, recompute each touched story's aggregates: source_count (distinct feeds), canonical headline/summary (take the highest-authority or earliest article), then re-score (§5c). Track which stories were created vs updated to emit the right WS event.

5c. Score (scoring.py) — rule-based, must sum to signalScore

Component caps chosen so totals land like the contract example (91 = 28+22+15+16+10):

Component	Range	Rule
sourceAuthority	0–30	sum of per-source authority weights (default 6; majors higher via config map), capped 30
freshness	0–25	round(25 * max(0, 1 - age_hours_of_newest / 48))
localRelevance	0–15	hits against a local lexicon (place names, currency, institutions) scaled, capped 15
crossSourceConfirmation	0–20	min(20, (distinct_sources - 1) * 5)
topicImportance	0–10	per-topic map: politics/finance=10, africa=8, tech=7, else 5

signalScore = clamp(sum, 0, 100).

• Topic classification: keyword lexicons per {finance, tech, politics, africa}; pick the highest-scoring; tie-break to africa. Keep lexicons in config.py.
• Consensus (set when ≥2 sources): e.g. "All N sources confirm the report. {SourceA} and {SourceB} agree." — template is fine for v1.
• Conflict (nullable, default null): v1 heuristic — set a generic line only if a headline contains dispute markers (deny, dispute, contradict, reject), else null. (Leave a TODO to upgrade with NLP/LLM later.)

5d. Staleness sweep (scheduler.py)

• Periodically (e.g. every 5 min), any non-stale Story with no new article in 24h → set is_stale=True, broadcast story.stale. Exclude stale stories from GET /stories by default.

5e. Pagination

Keyset cursor over (signal_score DESC, id DESC). nextCursor = base64 of "{signal_score}:{id}" of the last row; on after, decode and filter (signal_score < s) OR (signal_score == s AND id < id). hasMore = more rows exist; total = total non-stale stories (optionally filtered by topic/min_signal).

---

6. WebSocket /ws

On connect: accept(), register the socket. The server emits these (client refetches full objects via REST as needed):

{"type":"story.updated","storyId":"story_x","signalScore":94,"sourceCount":9,"updatedAt":"...Z"}
{"type":"story.created","storyId":"story_y","headline":"...","signalScore":42,"topic":"politics","createdAt":"...Z"}
{"type":"story.stale","storyId":"story_z","updatedAt":"...Z"}
{"type":"feed.health","feedId":"feed_a","health":"failing","failureCount":3,"updatedAt":"...Z"}

• Send {"type":"ping"} every 30s; the client replies {"type":"pong"} (you can ignore it). The client ALSO sends WebSocket-protocol pings — Uvicorn/websockets auto-answers those, no action needed.
• Keep a ConnectionManager with broadcast(dict); the ingest/correlate/scoring code calls it. Drop dead sockets on send failure.
• Reconnect/backoff is handled entirely client-side — just accept new connections.

---

7. Proxmox deployment (recommended: unprivileged LXC)

A Python web service is a perfect fit for an LXC container (lighter than a VM).

1. Create the container (Proxmox shell or UI):

   # Debian 12 template, on the Proxmox host:
   pveam update && pveam available | grep debian-12
   pveam download local debian-12-standard_*_amd64.tar.zst
   pct create 140 local:vztmpl/debian-12-standard_*_amd64.tar.zst \
     --hostname jarvis-api --cores 2 --memory 1024 --swap 512 \
     --net0 name=eth0,bridge=vmbr0,ip=dhcp \
     --rootfs local-lvm:8 --unprivileged 1 --features nesting=1
   pct start 140 && pct enter 140
   

   Note the container's DHCP IP (ip a) — that's the host the iPhone points at. A static IP / DHCP reservation is recommended so the app's saved host stays valid.

2. Inside the container:

   apt update && apt install -y python3 python3-venv python3-pip git
   adduser --system --group jarvis
   mkdir -p /opt/jarvis && chown jarvis:jarvis /opt/jarvis
   # copy the server/ folder here (git clone, scp, or pct push)
   cd /opt/jarvis
   python3 -m venv .venv && . .venv/bin/activate
   pip install -r requirements.txt
   

3. systemd service (/etc/systemd/system/jarvis.service):

   [Unit]
   Description=Jarvis news-correlation API
   After=network-online.target
   Wants=network-online.target
   
   [Service]
   User=jarvis
   WorkingDirectory=/opt/jarvis
   ExecStart=/opt/jarvis/.venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8080
   Restart=always
   RestartSec=3
   Environment=JARVIS_DB=/opt/jarvis/jarvis.db
   
   [Install]
   WantedBy=multi-user.target
   

   systemctl daemon-reload && systemctl enable --now jarvis
   systemctl status jarvis
   

4. Verify from the Proxmox host or your Mac (same LAN):

   curl http://<container-ip>:8080/api/v1/health
   curl http://<container-ip>:8080/api/v1/stories | head
   

5. Point the app at it: launch Jarvis on the iPhone/sim → onboarding → enter http://<container-ip>:8080 as <container-ip>:8080 (the app prepends http:// and ws:// itself — enter just host:port, e.g. 192.168.30.50:8080).

Networking notes: bind 0.0.0.0 (not 127.0.0.1) so the LAN can reach it. If the Proxmox host or container runs a firewall, allow TCP 8080. No TLS/auth by design — keep this on a trusted VLAN or behind the homelab firewall/VPN, per the contract.

---

8. Acceptance checklist (done = the app works end-to-end)

• GET /api/v1/health returns ok JSON; onboarding "Connect" succeeds.
• GET /api/v1/stories returns camelCase, ...Z dates, sorted by signalScore desc.
• Stories actually cluster multi-source articles (not 1 story per article).
• scoreBreakdown ints sum exactly to signalScore.
• GET /stories/{id} includes a chronological timeline[].
• GET /articles/{id} returns full body (offline reader works).
• GET /feeds shows health states; POST /feeds (201) and DELETE (204) work.
• WebSocket /ws connects; the app's header shows LIVE; a score change pushes story.updated and the feed reorders.
• Runs as a systemd service in the Proxmox LXC and survives a container reboot.

---

9. Hand-back

The iOS client is in ../Jarvis (SwiftUI, builds with xcodegen + Xcode 26 / iOS 17). While wiring the frontend I had to fix three bugs in the provided client layer that also document real expectations of your server — worth knowing:

• StoryStore patches stories in-place on story.updated using signalScore + sourceCount from the event (so those two fields are required on that event).
• The client treats HTTP http:// and ws:// (no TLS).
• On relaunch it reconnects REST+WS from the saved host, then immediately calls GET /stories — so the server should be ready to serve quickly after boot.

Questions for whoever runs this: which RSS feeds to seed by default, and the local lexicon (which country/region is "local" for localRelevance)? The contract examples are Uganda/East-Africa centric (Daily Monitor, NilePost, BoU) — default seed_feeds.py and the local lexicon to that unless told otherwise.