CodeResearch pipeline · validation
yorkline-pipeline
The research-to-publish pipeline behind The Yorkline: scrapers, de-duplication, school-name resolution, and a validator that fails the build if an article cites a source that isn't in the inventory.
- My role
- Author
- Built with
- TypeScript, Node.js
01Collectscrapers → candidates
02Mergededupe · resolve schools
03Selecteditorial overrides
04Validateevery cite in inventory
05Publishstatic build
build fails if any article cites a URL that isn't in the research inventory
Collect
Scrapers gather candidate stories into a research inventory.
Merge
Candidates are de-duplicated and school names resolved so the same event isn't reported twice.
Validate
Every article must cite URLs that exist in the inventory; the validator runs before each build and fails it on errors.
Keep exploring