Method

1. Architecture

1. Sources14 watched sources: 12 news searches and 2 official pages, covering ID, ASEAN, MY, SG, TH, VN, PH.
2. ScannerTwice a week, GitHub Actions runs npm run scan. It fetches, parses, scores and deduplicates.
3. TriageOptional. If an API key is set, Claude gives a second opinion on each new headline.
4. RadarNew leads are written to data/radar.yml and appear on the Radar page, marked unreviewed.
5. ArchiveEvery source without a preserved copy gets one in the Internet Archive, recorded next to the source.
6. AuthorThe author opens the source, judges it, and either dismisses the lead or writes an entry.
7. GatesEvery change passes the schemas, the validator, the tests and a full build.
8. OutcomesEach build computes every promise outcome as of that day and renders the receipts and their share cards.
9. SiteA commit to main triggers a Vercel build. The static site updates within minutes.

The design has one rule at its centre: automation may find things, but only the author may publish a claim. The scanner writes to the Radar and nowhere else, and the archiver only adds links to preserved copies beside sources. Entries, instruments and promises change only through a human edit that passes the same checks as every other change. Promise outcomes are never edited at all: they are computed.

2. Technology

3. Repository map

data/
  entries/*.yml        one file per entry (the ledger itself)
  instruments.yml      laws, drafts and guidance, with dossier fields
  deadlines.yml        public promises, newest first, in the promise format
  schema.json          JSON Schema for entries
  ontology.yml         institutions and curated, sourced links
  build-log.yml        every change asked for, shown below
  radar.yml            scanner output: unreviewed leads
  scan-log.yml         one row per scan
  watch/sources.yml    what the scanner reads
  watch/keywords.yml   how it scores
  watch/seen.json      ids already seen, so leads never return
docs/standard/v0.1/   the promise format: spec, JSON Schema, worked examples
kit/                   a starting point for a ledger in another country
scripts/
  validate.mjs         schemas and project rules, run before every build
  promises.mjs         deadline rules and computed outcomes, used everywhere
  archive.mjs          preserved copies of sources in the Internet Archive
  ontology.mjs         fuses all data into one graph of objects and links
  make-share-images.mjs  link preview cards and icons (resvg, fonts/)
  *.test.mjs           tests for promises, archive and ontology
  scan.mjs             the monitoring layer
  scan/parse.mjs       RSS, Atom and HTML link parsing
  scan/score.mjs       scoring, deduplication, dash rule
  scan/triage.mjs      optional Claude second opinion
  scan/scan.test.mjs   tests on synthetic fixtures
  new-entry.mjs        creates a dated entry from a template
  check-links.mjs      reports sources that stopped responding
site/                  Astro static site, no client framework
.github/workflows/
  ci.yml               tests, validation and build on every push
  scan.yml             the scheduled scan

4. Data model

• Entry (schema): id, date, instrument, status, evidence level, titles and summaries in both languages, the seven Safety lens codes with a basis sentence in both languages and the day they were checked, last checked date, EU comparison, the author's take (optional, written only by the author), and at least one source with title, outlet, language and URL. An optional quote must be under 15 words.
• Instrument: id, names, type, status, legal force, a source, and dossier fields: issuer, legal form, key points, EU counterpart, references, and whether the author has read it in full.
• Promise (format, schema): date said, country, who said it and their role, the claim type, for dated promises a deadline rule, deadline and precision, the wording behind the deadline, the promise and what happened since in both languages, an optional outcome date with its source, the evidence level, and the source with its archived copy. There is no outcome field: the outcome is computed.
• Analysis paper (rules): Markdown with front matter (title, author, published and facts-as-of days, version, sources file, summary), numbered sources with access dates and archived copies, and marked quotes. Bahasa Indonesia texts may be partial draft translations.
• Frontier incident (schema): developer, kind, the activity, awareness, notice and disclosure days as sources give them, who saw it first, the authority notified and the channel, countries, summaries in both languages and sources. Day counts are computed by scripts/frontier.mjs.
• Reading item (schema): title, authors, year, kind, url, status (planned, reading, read), and for items read a finish date and the author's notes in both languages.
• Lead: id (a hash of the normalised URL), date found, date published, headline, URL, outlet, country, language, source id, score, matched terms, an optional instrument hint, an optional Claude triage, and a review status: new, reviewed, promoted or dismissed.

5. The scanner

The scanner (scripts/scan.mjs) runs these steps in order:

1. Read the source list. 14 sources in data/watch/sources.yml. News searches use the Google News RSS endpoint in the right language edition, limited to the last 30 days. Official pages are read for their links.
2. Fetch. Each source is requested with a 25 second timeout and one retry, identifying itself as the Monsoon Ledger scanner. A failed source is logged and skipped. The run fails only if every source fails.
3. Parse. A small parser extracts headline, link, outlet, publication date and summary from RSS or Atom, or visible link text from HTML. It keeps only text and absolute http or https links, never markup or scripts.
4. Score. Each headline is matched against 38 weighted terms in data/watch/keywords.yml, in Bahasa Indonesia and English. The score is the sum of the weights of the distinct terms found. A lead is kept at a score of 4 or more, only if at least one core subject term (AI or data protection) is among them, so institution names alone never qualify, and unless one of 11 exclusion terms appears.
5. Deduplicate. URLs are normalised (host case, tracking parameters, fragments, trailing slashes) and hashed into a ten character id. A lead is skipped if its id was seen before, if the same headline arrived through another URL, or if the story is already a source in the ledger. Because news searches link through redirects, the ledger's own source URLs are also compared by their readable slug.
6. Hint and tag. Instrument hint terms suggest which instrument a lead may concern, linked on the Radar. The country tag comes from the headline, then the outlet name, and falls back to the source that found it, so a Malaysian story found by an Indonesian search is tagged Malaysia.
7. Triage, optional. With an ANTHROPIC_API_KEY secret set, up to 30 new leads go to Claude in one request. Claude sees only headline, outlet, date and country, and must answer in a fixed JSON schema: relevant or not, event type, instrument, a one sentence restatement of the headline, and confidence. Returned ids and instrument ids are checked against the input, summaries are capped at 30 words, and any error or refusal leaves the leads untriaged. On the site the summary is labelled as a machine summary of the headline.
8. Write. New leads are merged into data/radar.yml, newest first. Unreviewed leads are kept 180 days, reviewed ones a year, and the seen index 400 days so a pruned lead never comes back. Em and en dashes in headlines become hyphens, because the project bans them.

The scanner is tested on synthetic fixtures (scan.test.mjs): parsing, scoring, exclusion, deduplication, the dash rule, and that a second run adds nothing new.

6. Promises, receipts and cards

• One outcome module. scripts/promises.mjs derives deadlines from their rules and computes each promise's outcome (met, missed, open or no deadline), its days past or left, and the headline counts. The validator, the ontology, every page, the RSS feeds and the share cards all call it, so a number cannot differ between two places. Outcomes are computed in Western Indonesian Time and never stored.
• As of the build day. Every count states the day it was computed for. A small script in the page layout brings day counts on receipts up to the reader's day. If a deadline has passed since the build, the page keeps its build-day values, which agree with each other, until the next build, at least twice a week through the scan commit.
• Receipts. Each promise has a permanent page at /en/promises/<key>/ and /id/promises/<key>/, where the key is the date said and the speaker, the same key as its ontology object. It shows the wording in Bahasa Indonesia and English, the deadline and its basis, the computed outcome and day count, the evidence, the source and its archived copy, citations in plain text, APA and BibTeX with the data version (the short hash of the last commit that changed data/), and the entries and instruments it connects to.
• Share cards. scripts/make-share-images.mjs draws each card as SVG and renders it to PNG with resvg during the site build: one card per receipt per language, plus the site card. Each card shows its outcome, its day count and the day it was made. The same script regenerates the site icons. The full build takes seconds.
• Feeds. The RSS feeds get an item when a promise changes outcome, dated the day after a missed deadline or the day a promise was kept.
• Tests. promises.test.mjs covers leap years, month ends, months counted from the 31st, the deadline day itself, a promise kept on the deadline day, conditional and status claims, and the time zone.

7. Evidence archive

• What. scripts/archive.mjs (npm run archive) gives every source url in the promises, the instruments, the entries, the analysis sources and the frontier incident reference a preserved copy in the Internet Archive.
• How. It asks the Wayback Machine availability API for an existing snapshot first. If there is none, it requests one through Save Page Now, at most eight per run with a twelve second pause between requests.
• Where it writes. Only archive_url and archived_at, beside each source, by editing lines so comments and folded text stay exactly as they were. It never changes any other field.
• When. After every scheduled scan. It may fail without stopping the scan or the build; failures are logged in the workflow, and the validator warns about every source still without a copy.
• Tests. archive.test.mjs mocks the network and checks snapshot lookup, saving, rate limiting, failure handling, dry runs and that the file edits touch nothing else.

8. Schedule and deployment

• When. Every Monday and Thursday at 06:17 Western Indonesian Time (cron 17 23 * * 0,3 in UTC), so fresh leads are waiting at the start of the working day in Jakarta. Twice a week matches how often Indonesian AI policy news actually moves, without flooding the Radar.
• What runs. The scan workflow installs dependencies, runs the tests, runs the scan, archives sources, validates the data and builds the site. Only if the validation and the build pass does it commit radar.yml, scan-log.yml, seen.json and any new archive links to main.
• Publishing. The commit triggers Vercel, which builds and serves the new static site. No server is involved at any point.
• On pull requests. Changes to the scanner run it as a dry run against the live sources, without Claude and without committing, so a change is tested on real feeds before it is merged.
• On demand. The workflow can also be started by hand from the GitHub Actions tab.

9. Quality gates

Every push, by the author or by the scanner, passes four gates in CI: the tests (scanner, promises, ontology, archive, analysis papers, quote check, safety lens, frontier incidents, reading list and language parity), the validator, the JSON Schemas and a full site build, which ends with a parity check: every page must exist in English and Bahasa Indonesia unless it is marked partial. The validator also checks the analysis papers (every citation, every source, https only, no duplicate urls, quote length and one quote per source), the results of the quote check, the safety lens rules, the frontier incident reference and the reading list. npm run links checks every internal link and anchor in the built site. The validator checks entries, instruments, promises, the watched sources and every Radar lead. Promises are checked against the published promise schema, and every deadline is recomputed from its rule. The worked examples in the format's documentation are checked the same way. Sources without an archived copy are reported as warnings, never as failures. Any em or en dash anywhere in the project is rejected. With npm run validate -- --data <folder> the same validator checks only a promise ledger's data folder, such as the kit. The checks applied to entries are listed on the Research protocol page.

10. From lead to entry

1. Open the Radar and pick a lead.
2. Open the source, then look for the primary text or the official statement it reports, as the verification protocol requires.
3. If it does not hold up, set its status to dismissed in data/radar.yml. If it is worth watching, set reviewed.
4. If it holds up, run npm run new-entry -- --slug id-topic, fill every field from the sources you read, and set the lead to promoted with promoted_to pointing at the new entry id.
5. Run npm run validate, commit and push. CI checks it and Vercel publishes it.

11. How Claude was used

Monsoon Ledger was built from an empty repository with Claude Code, Anthropic's coding agent, in cloud sessions linked to the GitHub repository. The division of work is fixed:

• The author decides the scope, chooses what to track, reads the sources, sets every evidence level, writes every "My take", and approves every merge into main.
• Claude writes code and configuration, drafts and translates text for the author to review, checks URLs and legal citations against databases before they are used, and runs the tests, validator and build before each push.
• The rules live in CLAUDE.md, which Claude reads at the start of every session: never invent a fact, quote, date or URL, leave a marked TODO instead, never rewrite the author's own words, and pass the validator. The validator enforces the parts a machine can check.

Build so far. Every update the author asks for gets a row in data/build-log.yml, including fixes, reverts and rollbacks, so the history of the site stays visible. Newest first, each with its pull request and commits.

12. Keeping it current

• Twice a week, automatic. The scan runs, the Radar updates and the scan history records it.
• Weekly, the author. Review the Radar, dismiss or promote each new lead, and write entries for what holds up.
• When something happens to a promise. Set its outcome_date and outcome_source_url in data/deadlines.yml, update the one-line outcome text, and add a dated line to the changelog. The outcome itself is computed. A new promise gets a row with its claim type, deadline rule and the wording behind it.
• Monthly. Run npm run links, re-open the sources of entries listed under "due for re-checking" on the Radar, and update their last checked dates.
• When coverage drifts. Add or adjust sources and terms in data/watch/. The pull request dry run shows the effect on live feeds before merging.

13. Security and privacy

• No analytics, no cookies, no trackers and no external fonts. The site is static files.
• The only secret is the optional ANTHROPIC_API_KEY, stored as a GitHub Actions secret, never in the repository. Pull request dry runs never use it.
• The scanner workflow has write access to repository contents only, and only on scheduled or manual runs.
• Feeds and pages are treated as untrusted input: only text and links are extracted, and links open with rel="noopener nofollow".
• The scanner collects headlines from public news and official sites. It does not collect social media posts or data about private individuals.

14. Limits of automation

• Headlines only. The scanner reads headlines and feed summaries, not full articles. Relevance is judged on little text.
• Recall and precision. Keyword rules miss stories phrased in unexpected ways and let some noise through. The threshold trades one against the other, and the Radar shows the matched terms so the choice is visible.
• Google News links. News search results link through Google News, which redirects to the outlet. The outlet name is recorded so the original can be found.
• Official sites. Some government pages render content with JavaScript or block automated requests. Those sources fail visibly in the scan history rather than silently.
• Machine summaries. Claude's one line restates a headline. It is not verified and is labelled that way.
• Archive coverage. Some sites refuse the Internet Archive, and Save Page Now limits anonymous requests. Unarchived sources stay flagged until a later run succeeds.

15. Run it yourself

git clone https://github.com/SteveBehren/monsoon-ledger
cd monsoon-ledger
npm run setup          # install dependencies (Node 22.23.4 or newer)
npm test               # scanner, promise, ontology and archive tests
npm run validate       # schemas and project rules
npm run validate -- --data kit/data     # check a promise ledger folder only
npm run scan -- --dry-run --no-claude   # live scan, writes nothing
npm run archive -- --dry-run            # check archive coverage, writes nothing
node scripts/make-share-images.mjs      # regenerate the site icons
npm run dev            # local site at http://localhost:4321

Last scan recorded: 2026-11-05T03:29:09Z Radar →