An AI agent that does literature surveys is only as good as the libraries it knows to look in. This sounds obvious, but the operational consequence isn’t — every agent system that does any kind of research work converges on a stable set of sources it queries, in a stable order, with a stable ranking discipline — and the difference between an agent that produces useful surveys and one that hallucinates citations is almost entirely in this stack. The model layer matters less than you’d think; the source-stack layer matters more.
This field guide is the plain-language tour of that stack — for the engineer building an agent that does scientific literature work, the layers are arXiv (preprints), Semantic Scholar (citation graphs), Crossref (canonical metadata), OpenAlex (the open-graph alternative), GitHub (code), Papers With Code (benchmarks), YouTube and Spotify (lectures and conference talks). The guide walks through each one carefully — what it’s actually good for, how to query it, what the rate limits and quotas look like, the failure modes — and then assembles them into the pipeline a competent research agent runs.
It is the practical companion to the Skills in AI Agents and Knowledge & Memory in LLMs monographs — those documents cover the architecture layer; this one is about the specific data sources an agent in a research domain should be wired to.
What it covers
About nineteen minutes of reading. Designed for one-source-at-a-time reference use afterward.
§ 01 — Papers (arXiv, Semantic Scholar, Crossref, OpenAlex). The four sources that account for almost all serious academic-literature work. arXiv for the freshest preprints (most fields, daily updates). Semantic Scholar for the citation graph (who cites whom, who built on what). Crossref for canonical metadata (the DOI registry; the source of truth for “what this paper is”). OpenAlex for the open-graph alternative that’s been quietly eating into the commercial scientific-graph market.
§ 02 — Code (GitHub, Papers With Code). GitHub for the implementations themselves. Papers With Code for the bridge between papers and the code that goes with them. The query patterns and the data-freshness considerations.
§ 03 — Video & other (YouTube, Spotify, podcasts). The non-paper sources that increasingly carry the most current thinking — recorded conference talks, lab seminars, working-paper-stage discussions. The transcription-and-indexing pipeline that turns these into searchable text.
§ 04 — The pipeline. The order an agent actually queries these in. Why papers come first, then code, then video. The deduplication step (the same paper appears in arXiv, Semantic Scholar, Crossref, and OpenAlex — only one of those should be canonical for the agent’s index). The merge step.
§ 05 — Ranking & retrieval. Once you have results from across the stack, how do you rank them? Recency. Citation count. Author authority. Source-trust score. The ranking function that decides what gets into the agent’s context window.
§ 06 — Quotas & limits. The rate-limit table for each source. The exponential-backoff patterns. The user-agent / contact-email discipline that keeps you on the right side of the libraries’ terms of service.
§ 07 — Failure modes. The dozen ways this pipeline breaks. Hallucinated citations. Stale metadata. Out-of-date Papers-With-Code links. Authors with ambiguous name disambiguation. The defensive patterns.
Downloads
The harness kit and the mcp-builder skill bundle live at the end of the field guide. Both are also linked directly here:
↓ research-harness.zip — a working harness with a literature-survey skill, a person-dossier skill, a scientific-researcher agent profile, and the FastMCP server that exposes the source stack as tools.
↓ mcp-builder-skill.zip — a standalone Skill that teaches an agent how to build new MCP servers (FastMCP for Python, TypeScript SDK reference, evaluation discipline).
Both kits are MIT-licensed and ready to drop into a project.
Read it
The field guide lives at its own URL with a warm-paper editorial layout — teal accent for papers, rust for code, gold for video. Seven sections, a downloadable harness, and the MCP-builder skill kit.
← Back to Autonomy