Initial commit: Jarvis iOS app
SwiftUI client for a self-hosted RSS news-correlation platform: signal feed, story detail, article reader, feed manager, and LAN⇄Tailscale connectivity. Project generated from project.yml via XcodeGen. Includes CI build matrix (macOS 14/15 × Debug/Release), issue templates, backlog, and API/backend handoff docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
336
docs/backend/BACKEND_HANDOFF.md
Normal file
336
docs/backend/BACKEND_HANDOFF.md
Normal file
@@ -0,0 +1,336 @@
|
||||
# Jarvis Backend — Build & Deploy Handoff (for Codex on Proxmox)
|
||||
|
||||
You are building the **server side** of Jarvis: a self-hosted RSS news-correlation
|
||||
platform. A finished SwiftUI iOS client already exists and will connect to you
|
||||
**unmodified** — so the API shapes below are not suggestions, they are a contract.
|
||||
The canonical spec is `API/contract.md` in this repo; this document adds the
|
||||
implementation plan, the algorithms, the client-compat gotchas, and the Proxmox
|
||||
deployment.
|
||||
|
||||
**Stack (already decided):** Python 3.11+ · FastAPI · Uvicorn · SQLite (via SQLModel) ·
|
||||
scikit-learn TF-IDF for correlation · rule-based scoring. No external API keys.
|
||||
No auth (local network / self-hosted only).
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR of what to build
|
||||
|
||||
A single FastAPI app that does four jobs:
|
||||
|
||||
1. **Ingest** — poll RSS feeds on a schedule, parse articles, track feed health.
|
||||
2. **Correlate** — cluster articles from different sources into *Stories* (TF-IDF + cosine).
|
||||
3. **Score** — compute a server-side `signalScore` (5 components) + topic + consensus/conflict.
|
||||
4. **Serve** — the REST API under `/api/v1` and a WebSocket at `/ws`, broadcasting live events.
|
||||
|
||||
Then deploy it as a systemd service inside a Proxmox **LXC container**, bound to
|
||||
`0.0.0.0:8080`, reachable from the iPhone on the LAN.
|
||||
|
||||
---
|
||||
|
||||
## 1. NON-NEGOTIABLE client-compatibility rules
|
||||
|
||||
The iOS client decodes JSON with `keyDecodingStrategy = .convertFromSnakeCase`
|
||||
and `dateDecodingStrategy = .iso8601`, and it **silently drops any object that
|
||||
fails to decode**. So:
|
||||
|
||||
1. **Dates must be exactly `YYYY-MM-DDTHH:MM:SSZ`** — UTC, the literal `Z`, and
|
||||
**no fractional seconds**. Pydantic's default (`...+00:00` with microseconds)
|
||||
will FAIL to decode and stories will vanish. Force the format with a serializer
|
||||
(see §4).
|
||||
2. **JSON keys: camelCase** (`signalScore`, `sourceCount`, `pollIntervalSeconds`,
|
||||
`publishedAt`, `nextCursor`, `hasMore`, …). FastAPI: set field aliases via
|
||||
`alias_generator=to_camel`, serialize with `response_model_by_alias=True` (default).
|
||||
3. **Exact query-param names** the client sends to `GET /stories`:
|
||||
`limit`, `after` (the cursor), `topic`, `min_signal`. Read those names verbatim.
|
||||
4. **`DELETE /feeds/:id` must return HTTP 204** (no body). The client checks for 204.
|
||||
5. **`POST /feeds` returns 201** with the created feed object. The client sends
|
||||
`{"url": ..., "name": ..., "pollIntervalSeconds": ...}` — accept camelCase.
|
||||
6. **Base URL is `http://<host>/api/v1`** and **WebSocket is `ws://<host>/ws`** —
|
||||
the client builds these strings literally, so REST lives under `/api/v1` and the
|
||||
socket at the root path `/ws`. Plain `http`/`ws` (not TLS) is expected.
|
||||
7. **No CORS needed** (native app, not a browser), but harmless to enable `*`.
|
||||
8. Error responses use `{"error": {"code", "message", "status"}}` (see contract §Error).
|
||||
|
||||
---
|
||||
|
||||
## 2. Endpoints (all under `/api/v1` unless noted)
|
||||
|
||||
Full example payloads are in `API/contract.md`. Summary:
|
||||
|
||||
| Method | Path | Returns | Notes |
|
||||
|---|---|---|---|
|
||||
| GET | `/health` | `{status, version, storiesCount, feedsCount, uptime}` | client calls on launch to validate connection |
|
||||
| GET | `/stories` | `{data[], nextCursor, hasMore, total}` | sorted by `signalScore` desc; params `limit`(≤100, def 20), `after`, `topic`, `min_signal` |
|
||||
| GET | `/stories/{id}` | `StoryDetail` (adds `timeline[]`) | 404 `story_not_found` if missing |
|
||||
| GET | `/articles/{id}` | `Article` (full body for offline cache) | 404 if missing |
|
||||
| GET | `/feeds` | `{data: [Feed]}` | all feeds + health |
|
||||
| POST | `/feeds` | `Feed` (201) | body `{url, name, pollIntervalSeconds?}`; 400 `invalid_url` if not a real feed |
|
||||
| DELETE | `/feeds/{id}` | 204 | 404 `feed_not_found` if missing |
|
||||
| WS | `/ws` (root, not `/api/v1`) | event stream | see §6 |
|
||||
|
||||
### Object shapes (camelCase, dates as `...Z`)
|
||||
|
||||
- **StorySummary**: `id, headline, summary, topic, signalScore, scoreBreakdown,
|
||||
sourceCount, sources[], consensus, conflict, updatedAt, createdAt`
|
||||
- **scoreBreakdown**: `{sourceAuthority, freshness, localRelevance,
|
||||
crossSourceConfirmation, topicImportance}` (ints, sum == `signalScore`)
|
||||
- **StorySource**: `{id, name, url, publishedAt, isBreaking}`
|
||||
- **StoryDetail**: StorySummary fields **minus `sources`**, **plus** `timeline[]`
|
||||
- **TimelineEntry**: `{articleId, source, headline, publishedAt, isBreaking}`
|
||||
- **Article**: `{id, storyId, source, sourceUrl, headline, body, imageUrl, author, publishedAt}`
|
||||
- **Feed**: `{id, name, url, health, pollIntervalSeconds, failureCount, lastFetchedAt, articleCountToday}`
|
||||
- `health ∈ {"active","failing","dead"}`
|
||||
- **topic** slugs used by the app's filter pills: `finance`, `tech`, `politics`, `africa`
|
||||
(classify everything into one of these; fall back to the best match).
|
||||
|
||||
---
|
||||
|
||||
## 3. Suggested project layout
|
||||
|
||||
```
|
||||
server/
|
||||
app/
|
||||
main.py # FastAPI app, lifespan: init DB, seed, start poller
|
||||
config.py # tunables: thresholds, lexicons, source-authority map
|
||||
db.py # SQLModel engine/session, init_db()
|
||||
models.py # Feed, Article, Story (SQLModel tables)
|
||||
schemas.py # Pydantic response models (camelCase aliases + Z dates)
|
||||
ingest.py # RSS poll + parse + feed-health state machine
|
||||
correlate.py # TF-IDF clustering: assign articles -> stories
|
||||
scoring.py # signal-score components, topic classify, consensus/conflict
|
||||
events.py # WebSocket ConnectionManager + broadcast + 30s ping loop
|
||||
scheduler.py # asyncio loop: poll due feeds, correlate, score, sweep stale
|
||||
routers/
|
||||
stories.py articles.py feeds.py health.py ws.py
|
||||
seed_feeds.py # default feed list inserted on first run
|
||||
requirements.txt # already in this folder
|
||||
jarvis.service # systemd unit (see §8)
|
||||
README.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Data model (SQLite via SQLModel)
|
||||
|
||||
- **Feed**: `id (pk, "feed_"+short uuid)`, `name`, `url`, `health`,
|
||||
`poll_interval_seconds`, `failure_count`, `last_fetched_at (nullable)`, `created_at`.
|
||||
`articleCountToday` is **computed** at query time (count of this feed's articles
|
||||
with `created_at >= today 00:00 UTC`).
|
||||
- **Article**: `id ("art_"+uuid)`, `feed_id (fk)`, `story_id (fk, nullable)`,
|
||||
`source` (feed name), `source_url`, `headline`, `body`, `image_url (nullable)`,
|
||||
`author (nullable)`, `published_at`, `guid (unique — dedup key)`, `created_at`,
|
||||
`match_text` (lowercased headline + body, for TF-IDF).
|
||||
- **Story**: `id ("story_"+uuid)`, `headline`, `summary`, `topic`, `signal_score`,
|
||||
the 5 breakdown ints, `consensus (nullable)`, `conflict (nullable)`,
|
||||
`source_count`, `is_stale (bool)`, `created_at`, `updated_at`.
|
||||
|
||||
Store **all datetimes as naive UTC**. Serialize with this exact helper so the
|
||||
client's `.iso8601` accepts them:
|
||||
|
||||
```python
|
||||
from datetime import datetime, timezone
|
||||
from pydantic import field_serializer
|
||||
|
||||
def iso_z(dt: datetime) -> str:
|
||||
if dt.tzinfo is not None:
|
||||
dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
|
||||
return dt.strftime("%Y-%m-%dT%H:%M:%SZ") # NO microseconds, literal Z
|
||||
```
|
||||
|
||||
Apply via `@field_serializer("publishedAt","updatedAt","createdAt","lastFetchedAt")`
|
||||
returning `iso_z(value)` (return `None` for null `lastFetchedAt`).
|
||||
|
||||
Camel aliases:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
from pydantic.alias_generators import to_camel
|
||||
|
||||
class Schema(BaseModel):
|
||||
model_config = ConfigDict(alias_generator=to_camel, populate_by_name=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. The algorithms
|
||||
|
||||
### 5a. Ingest (`ingest.py`)
|
||||
- Fetch each due feed with `httpx.AsyncClient` (timeout ~10s), parse bytes with
|
||||
`feedparser` (run in `asyncio.to_thread`, it's blocking).
|
||||
- For each entry: dedup by `guid` (entry.id or link). Extract `headline` (title),
|
||||
`body` (richest of `content[].value` / `summary`, HTML-stripped with BeautifulSoup),
|
||||
`image_url` (media:content / enclosure / first `<img>`), `author`, `published_at`
|
||||
(`entry.published_parsed` → UTC; fall back to now).
|
||||
- **Feed-health state machine** on each poll:
|
||||
- success → `health="active"`, `failure_count=0`, `last_fetched_at=now`.
|
||||
- failure → `failure_count += 1`; `failing` at ≥1, `dead` at ≥5 (tunable).
|
||||
- On any health **change**, broadcast a `feed.health` WS event.
|
||||
|
||||
### 5b. Correlate (`correlate.py`)
|
||||
Goal: same real-world event across sources → one Story; stable Story `id`s.
|
||||
- Consider only articles from a rolling window (e.g. last **72h**).
|
||||
- Single `TfidfVectorizer` fit per cycle over (existing recent stories' aggregate
|
||||
text + the new unclustered articles' `match_text`). English stopwords, `ngram_range=(1,2)`.
|
||||
- **Greedy online assignment** per new article (process oldest→newest):
|
||||
- cosine-compare against current cluster representatives;
|
||||
- if `max_sim >= 0.22` (tunable) **and** within the time window → join that story;
|
||||
- else → create a **new** Story seeded by this article (append its vector as a new rep).
|
||||
- After assignment, recompute each touched story's aggregates: `source_count`
|
||||
(distinct feeds), canonical `headline`/`summary` (take the highest-authority or
|
||||
earliest article), then re-score (§5c). Track which stories were **created** vs
|
||||
**updated** to emit the right WS event.
|
||||
|
||||
### 5c. Score (`scoring.py`) — rule-based, must sum to `signalScore`
|
||||
Component caps chosen so totals land like the contract example (91 = 28+22+15+16+10):
|
||||
|
||||
| Component | Range | Rule |
|
||||
|---|---|---|
|
||||
| `sourceAuthority` | 0–30 | sum of per-source authority weights (default 6; majors higher via config map), capped 30 |
|
||||
| `freshness` | 0–25 | `round(25 * max(0, 1 - age_hours_of_newest / 48))` |
|
||||
| `localRelevance` | 0–15 | hits against a local lexicon (place names, currency, institutions) scaled, capped 15 |
|
||||
| `crossSourceConfirmation` | 0–20 | `min(20, (distinct_sources - 1) * 5)` |
|
||||
| `topicImportance` | 0–10 | per-topic map: politics/finance=10, africa=8, tech=7, else 5 |
|
||||
|
||||
`signalScore = clamp(sum, 0, 100)`.
|
||||
|
||||
- **Topic classification**: keyword lexicons per `{finance, tech, politics, africa}`;
|
||||
pick the highest-scoring; tie-break to `africa`. Keep lexicons in `config.py`.
|
||||
- **Consensus** (set when ≥2 sources): e.g. `"All N sources confirm the report.
|
||||
{SourceA} and {SourceB} agree."` — template is fine for v1.
|
||||
- **Conflict** (nullable, default null): v1 heuristic — set a generic line only if a
|
||||
headline contains dispute markers (`deny`, `dispute`, `contradict`, `reject`),
|
||||
else `null`. (Leave a TODO to upgrade with NLP/LLM later.)
|
||||
|
||||
### 5d. Staleness sweep (`scheduler.py`)
|
||||
- Periodically (e.g. every 5 min), any non-stale Story with **no new article in 24h**
|
||||
→ set `is_stale=True`, broadcast `story.stale`. **Exclude stale stories** from
|
||||
`GET /stories` by default.
|
||||
|
||||
### 5e. Pagination
|
||||
Keyset cursor over `(signal_score DESC, id DESC)`. `nextCursor` = base64 of
|
||||
`"{signal_score}:{id}"` of the last row; on `after`, decode and filter
|
||||
`(signal_score < s) OR (signal_score == s AND id < id)`. `hasMore` = more rows exist;
|
||||
`total` = total non-stale stories (optionally filtered by topic/min_signal).
|
||||
|
||||
---
|
||||
|
||||
## 6. WebSocket `/ws`
|
||||
|
||||
On connect: `accept()`, register the socket. The server emits these (client refetches
|
||||
full objects via REST as needed):
|
||||
|
||||
```jsonc
|
||||
{"type":"story.updated","storyId":"story_x","signalScore":94,"sourceCount":9,"updatedAt":"...Z"}
|
||||
{"type":"story.created","storyId":"story_y","headline":"...","signalScore":42,"topic":"politics","createdAt":"...Z"}
|
||||
{"type":"story.stale","storyId":"story_z","updatedAt":"...Z"}
|
||||
{"type":"feed.health","feedId":"feed_a","health":"failing","failureCount":3,"updatedAt":"...Z"}
|
||||
```
|
||||
|
||||
- Send `{"type":"ping"}` every **30s**; the client replies `{"type":"pong"}` (you can
|
||||
ignore it). The client ALSO sends WebSocket-protocol pings — Uvicorn/websockets
|
||||
auto-answers those, no action needed.
|
||||
- Keep a `ConnectionManager` with `broadcast(dict)`; the ingest/correlate/scoring code
|
||||
calls it. Drop dead sockets on send failure.
|
||||
- Reconnect/backoff is handled entirely client-side — just accept new connections.
|
||||
|
||||
---
|
||||
|
||||
## 7. Proxmox deployment (recommended: unprivileged LXC)
|
||||
|
||||
A Python web service is a perfect fit for an **LXC container** (lighter than a VM).
|
||||
|
||||
1. **Create the container** (Proxmox shell or UI):
|
||||
```bash
|
||||
# Debian 12 template, on the Proxmox host:
|
||||
pveam update && pveam available | grep debian-12
|
||||
pveam download local debian-12-standard_*_amd64.tar.zst
|
||||
pct create 140 local:vztmpl/debian-12-standard_*_amd64.tar.zst \
|
||||
--hostname jarvis-api --cores 2 --memory 1024 --swap 512 \
|
||||
--net0 name=eth0,bridge=vmbr0,ip=dhcp \
|
||||
--rootfs local-lvm:8 --unprivileged 1 --features nesting=1
|
||||
pct start 140 && pct enter 140
|
||||
```
|
||||
Note the container's DHCP IP (`ip a`) — that's the host the iPhone points at. A
|
||||
**static IP / DHCP reservation** is recommended so the app's saved host stays valid.
|
||||
|
||||
2. **Inside the container:**
|
||||
```bash
|
||||
apt update && apt install -y python3 python3-venv python3-pip git
|
||||
adduser --system --group jarvis
|
||||
mkdir -p /opt/jarvis && chown jarvis:jarvis /opt/jarvis
|
||||
# copy the server/ folder here (git clone, scp, or pct push)
|
||||
cd /opt/jarvis
|
||||
python3 -m venv .venv && . .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. **systemd service** (`/etc/systemd/system/jarvis.service`):
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Jarvis news-correlation API
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=jarvis
|
||||
WorkingDirectory=/opt/jarvis
|
||||
ExecStart=/opt/jarvis/.venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8080
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
Environment=JARVIS_DB=/opt/jarvis/jarvis.db
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
```bash
|
||||
systemctl daemon-reload && systemctl enable --now jarvis
|
||||
systemctl status jarvis
|
||||
```
|
||||
|
||||
4. **Verify from the Proxmox host or your Mac (same LAN):**
|
||||
```bash
|
||||
curl http://<container-ip>:8080/api/v1/health
|
||||
curl http://<container-ip>:8080/api/v1/stories | head
|
||||
```
|
||||
|
||||
5. **Point the app at it:** launch Jarvis on the iPhone/sim → onboarding → enter
|
||||
`http://<container-ip>:8080` **as `<container-ip>:8080`** (the app prepends
|
||||
`http://` and `ws://` itself — enter just `host:port`, e.g. `192.168.30.50:8080`).
|
||||
|
||||
> Networking notes: bind `0.0.0.0` (not `127.0.0.1`) so the LAN can reach it. If the
|
||||
> Proxmox host or container runs a firewall, allow TCP **8080**. No TLS/auth by design —
|
||||
> keep this on a trusted VLAN or behind the homelab firewall/VPN, per the contract.
|
||||
|
||||
---
|
||||
|
||||
## 8. Acceptance checklist (done = the app works end-to-end)
|
||||
|
||||
- [ ] `GET /api/v1/health` returns ok JSON; onboarding "Connect" succeeds.
|
||||
- [ ] `GET /api/v1/stories` returns camelCase, `...Z` dates, sorted by `signalScore` desc.
|
||||
- [ ] Stories actually cluster multi-source articles (not 1 story per article).
|
||||
- [ ] `scoreBreakdown` ints sum exactly to `signalScore`.
|
||||
- [ ] `GET /stories/{id}` includes a chronological `timeline[]`.
|
||||
- [ ] `GET /articles/{id}` returns full `body` (offline reader works).
|
||||
- [ ] `GET /feeds` shows health states; `POST /feeds` (201) and `DELETE` (204) work.
|
||||
- [ ] WebSocket `/ws` connects; the app's header shows **LIVE**; a score change pushes
|
||||
`story.updated` and the feed reorders.
|
||||
- [ ] Runs as a systemd service in the Proxmox LXC and survives a container reboot.
|
||||
|
||||
---
|
||||
|
||||
## 9. Hand-back
|
||||
|
||||
The iOS client is in `../Jarvis` (SwiftUI, builds with `xcodegen` + Xcode 26 / iOS 17).
|
||||
While wiring the frontend I had to fix three bugs in the provided client layer that
|
||||
also document real expectations of your server — worth knowing:
|
||||
- `StoryStore` patches stories in-place on `story.updated` using `signalScore` +
|
||||
`sourceCount` from the event (so those two fields are required on that event).
|
||||
- The client treats **HTTP `http://`** and **`ws://`** (no TLS).
|
||||
- On relaunch it reconnects REST+WS from the saved host, then immediately calls
|
||||
`GET /stories` — so the server should be ready to serve quickly after boot.
|
||||
|
||||
Questions for whoever runs this: which RSS feeds to seed by default, and the local
|
||||
lexicon (which country/region is "local" for `localRelevance`)? The contract examples
|
||||
are Uganda/East-Africa centric (Daily Monitor, NilePost, BoU) — default `seed_feeds.py`
|
||||
and the local lexicon to that unless told otherwise.
|
||||
```
|
||||
150
docs/backend/CODEX_PREHANDOVER_CHECKLIST.md
Normal file
150
docs/backend/CODEX_PREHANDOVER_CHECKLIST.md
Normal file
@@ -0,0 +1,150 @@
|
||||
# Jarvis Backend — Pre-Handover Checklist (Codex → integration)
|
||||
|
||||
Run every item before declaring the backend done. Most have a command + expected
|
||||
result, so "done" is **verified**, not assumed. Set the host once:
|
||||
|
||||
```bash
|
||||
HOST=http://<container-ip>:8080 # e.g. http://192.168.30.50:8080
|
||||
```
|
||||
|
||||
> The single biggest cause of a "connected but empty" app is **date format** and
|
||||
> **key casing** (§A). If §A doesn't pass, nothing else matters — the client drops
|
||||
> the objects silently.
|
||||
|
||||
---
|
||||
|
||||
## A. Contract compatibility (the silent killers)
|
||||
|
||||
- [ ] **Dates are `YYYY-MM-DDTHH:MM:SSZ`** — UTC, literal `Z`, NO fractional seconds.
|
||||
```bash
|
||||
curl -s "$HOST/api/v1/stories" | python3 -c '
|
||||
import sys,json,re
|
||||
d=json.load(sys.stdin)["data"]
|
||||
bad=[s[k] for s in d for k in ("updatedAt","createdAt")
|
||||
if not re.fullmatch(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z", s[k])]
|
||||
print("DATES OK" if not bad else ("BAD DATES: "+str(bad[:3])))'
|
||||
```
|
||||
- [ ] **Keys are camelCase** — `signalScore`, `scoreBreakdown`, `sourceCount`,
|
||||
`pollIntervalSeconds`, `publishedAt`, `nextCursor`, `hasMore`, `isBreaking`,
|
||||
`sourceUrl`, `imageUrl`, `storyId`, `articleCountToday`, `lastFetchedAt`.
|
||||
No snake_case anywhere in any response body.
|
||||
- [ ] **`scoreBreakdown` ints sum exactly to `signalScore`.**
|
||||
```bash
|
||||
curl -s "$HOST/api/v1/stories" | python3 -c '
|
||||
import sys,json
|
||||
d=json.load(sys.stdin)["data"]
|
||||
bad=[s["id"] for s in d if sum(s["scoreBreakdown"].values())!=s["signalScore"]]
|
||||
print("SCORES OK" if not bad else "SUM MISMATCH: "+str(bad))'
|
||||
```
|
||||
- [ ] **Stories sorted by `signalScore` descending** in `GET /stories`.
|
||||
- [ ] **Topic is always one of** `finance | tech | politics | africa` (the app's pills).
|
||||
- [ ] Unknown/extra fields are fine, but **no required field is missing or null**
|
||||
where the client expects a value (e.g. `headline`, `topic`, `signalScore`,
|
||||
`sourceCount`, `scoreBreakdown`, `updatedAt`, `createdAt`, `sources`).
|
||||
|
||||
## B. Endpoints behave exactly as the client calls them
|
||||
|
||||
- [ ] `GET /api/v1/health` → `{status,version,storiesCount,feedsCount,uptime}` (200).
|
||||
- [ ] `GET /api/v1/stories?limit=20` honors `limit` (≤100), returns
|
||||
`{data,nextCursor,hasMore,total}`.
|
||||
- [ ] **Pagination round-trips:** take `nextCursor`, pass it as `after`, get the next
|
||||
page with no overlap and no dupes.
|
||||
```bash
|
||||
C=$(curl -s "$HOST/api/v1/stories?limit=2" | python3 -c 'import sys,json;print(json.load(sys.stdin)["nextCursor"])')
|
||||
curl -s "$HOST/api/v1/stories?limit=2&after=$C" | python3 -c 'import sys,json;print("page2 ids:",[s["id"] for s in json.load(sys.stdin)["data"]])'
|
||||
```
|
||||
- [ ] `GET /api/v1/stories?topic=finance` filters; `&min_signal=80` filters.
|
||||
- [ ] `GET /api/v1/stories/{id}` → `StoryDetail` **with `timeline[]`** (chronological).
|
||||
- [ ] Bad id → **404** with `{"error":{"code":"story_not_found","message":...,"status":404}}`.
|
||||
- [ ] `GET /api/v1/articles/{id}` → full `body` present (not truncated to the RSS blurb
|
||||
if avoidable); `storyId` matches.
|
||||
- [ ] `GET /api/v1/feeds` → `{data:[Feed]}` with `health`, `failureCount`,
|
||||
`articleCountToday`, `lastFetchedAt`.
|
||||
- [ ] `POST /api/v1/feeds` with `{"url","name","pollIntervalSeconds"}` → **201** + feed body.
|
||||
```bash
|
||||
curl -s -X POST "$HOST/api/v1/feeds" -H 'Content-Type: application/json' \
|
||||
-d '{"url":"https://feeds.bbci.co.uk/news/world/rss.xml","name":"BBC World","pollIntervalSeconds":900}' -i | head -1
|
||||
```
|
||||
- [ ] Invalid feed URL → **400** `invalid_url`.
|
||||
- [ ] `DELETE /api/v1/feeds/{id}` → **204** no body; deleting missing id → 404 `feed_not_found`.
|
||||
|
||||
## C. Correlation & scoring are actually working
|
||||
|
||||
- [ ] Articles from **different sources** about the same event land in **one** Story
|
||||
(spot-check: a Story with `sourceCount >= 2` whose `timeline` shows ≥2 distinct sources).
|
||||
- [ ] Not the degenerate case of **one Story per article** (clustering threshold too high)
|
||||
or **everything in one Story** (threshold too low).
|
||||
- [ ] `sources[]` / `timeline[]` list the right outlets; exactly one entry flagged the
|
||||
earliest/`isBreaking` per the seed logic.
|
||||
- [ ] `consensus` text appears when `sourceCount >= 2`; `conflict` is `null` unless a
|
||||
real dispute marker is present.
|
||||
- [ ] `freshness` decays over time (an old Story scores lower than a fresh one with the
|
||||
same sources).
|
||||
|
||||
## D. WebSocket `/ws`
|
||||
|
||||
- [ ] `ws://<host>/ws` accepts a connection (test below).
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
import asyncio, websockets, json
|
||||
async def main():
|
||||
async with websockets.connect("ws://<container-ip>:8080/ws") as ws:
|
||||
print("connected; waiting for an event/ping (15s)...")
|
||||
try: print(await asyncio.wait_for(ws.recv(), 15))
|
||||
except asyncio.TimeoutError: print("no message in 15s (ok if idle)")
|
||||
asyncio.run(main())
|
||||
PY
|
||||
```
|
||||
- [ ] Server sends `{"type":"ping"}` ~every 30s and tolerates the client's
|
||||
`{"type":"pong"}` reply.
|
||||
- [ ] A score change broadcasts `story.updated` **with `signalScore` AND `sourceCount`**
|
||||
(the client requires both on that event).
|
||||
- [ ] A brand-new cluster broadcasts `story.created`; a stale one broadcasts `story.stale`.
|
||||
- [ ] A feed health transition broadcasts `feed.health` with `feedId`, `health`,
|
||||
`failureCount`, `updatedAt`.
|
||||
- [ ] Multiple simultaneous clients all receive broadcasts; a dropped client doesn't
|
||||
crash the loop.
|
||||
|
||||
## E. Ingest robustness
|
||||
|
||||
- [ ] A dead/unreachable feed URL drives `active → failing → dead` and increments
|
||||
`failureCount` without crashing the poller.
|
||||
- [ ] Duplicate articles (same guid/link re-published) are **not** inserted twice.
|
||||
- [ ] HTML is stripped from `body`; `imageUrl` is populated when the feed provides one.
|
||||
- [ ] Missing `published_at` falls back sanely (now) rather than erroring.
|
||||
- [ ] Poller respects each feed's `pollIntervalSeconds` (doesn't hammer sources).
|
||||
|
||||
## F. Deployment on Proxmox
|
||||
|
||||
- [ ] Runs under **systemd** (`systemctl status jarvis` = active/running), not a stray shell.
|
||||
- [ ] Binds **`0.0.0.0:8080`** (reachable from the LAN, not just localhost):
|
||||
`curl http://<container-ip>:8080/api/v1/health` works **from your Mac**, not only inside the container.
|
||||
- [ ] **Survives reboot:** `reboot` the LXC, then `curl .../health` succeeds with the
|
||||
service back up and data intact.
|
||||
- [ ] SQLite DB persists at a fixed path (e.g. `/opt/jarvis/jarvis.db`) and isn't wiped on restart.
|
||||
- [ ] Container has a **static IP / DHCP reservation** so the app's saved host stays valid.
|
||||
- [ ] Firewall (if any) allows inbound TCP **8080**.
|
||||
- [ ] Logs are reachable: `journalctl -u jarvis -n 50` shows clean startup + poll cycles.
|
||||
|
||||
## G. End-to-end with the real app (the real acceptance)
|
||||
|
||||
- [ ] Onboarding: enter `<container-ip>:8080` → **Connect succeeds**.
|
||||
- [ ] Home feed populates, ranked by signal, fade rendering looks right.
|
||||
- [ ] Header shows **LIVE** (green) — WebSocket connected.
|
||||
- [ ] Tap a story → detail with timeline → tap an article → reader shows full body.
|
||||
- [ ] Pull-to-refresh works; opening an article caches it (green dot / offline badge appears).
|
||||
- [ ] Feed manager (radio icon, top-right) lists feeds with correct health dots; add + swipe-delete work.
|
||||
|
||||
## H. Handover packet (give these back for integration)
|
||||
|
||||
- [ ] The **container IP/host** and confirmation the service is enabled on boot.
|
||||
- [ ] One **sample `GET /stories` response** and one **`GET /stories/{id}`** (so I can
|
||||
diff against the client decoder).
|
||||
- [ ] The **seed feed list** used and the **`localRelevance` region/lexicon** chosen.
|
||||
- [ ] Any deviations from `BACKEND_HANDOFF.md` (endpoints, fields, thresholds) called out.
|
||||
- [ ] `requirements.txt` frozen (`pip freeze`) and the repo/commit deployed.
|
||||
|
||||
---
|
||||
|
||||
**Definition of done:** §A–§F all green, §G walks through on a real device, and §H is
|
||||
handed back. At that point ping me and I'll run the client-side integration pass.
|
||||
8
docs/backend/requirements.txt
Normal file
8
docs/backend/requirements.txt
Normal file
@@ -0,0 +1,8 @@
|
||||
fastapi==0.115.6
|
||||
uvicorn[standard]==0.34.0
|
||||
sqlmodel==0.0.22
|
||||
feedparser==6.0.11
|
||||
httpx==0.28.1
|
||||
scikit-learn==1.6.1
|
||||
numpy==2.2.1
|
||||
beautifulsoup4==4.12.3
|
||||
Reference in New Issue
Block a user