dose.wiki — codebase overview
Next.js 16 + Convex + AI editorial pipelines — written for someone new to the repository
dose.wiki is a harm-reduction encyclopedia: a public website of substance articles, subjective-effect documentation, and trip reports. The repository contains three systems. The public Next.js site renders content. The login-protected /dev editor is where editors write and cite that content. Command-line pipelines scrape, generate, and migrate it. All three read and write the same Convex database, which is the source of truth.
01 The big picture in numbers
- ~580
- Substance articles in Convex
- ~230
- Subjective effects
- ~160
- Trip reports
- ~900
- TS/TSX files in
src/ - ~400
- Pipeline scripts in
scripts/
Roughly half of this repository is not the website. The scripts/ directory is nearly as large as the app itself, because pipelines produce most content. Scripts scrape content from source wikis. Claude generates prose through OpenRouter. A citation workflow ties claims to real literature, and a person reviews everything before publication. If you read only src/, you see the site but not the tooling that fills it.
02 System overview
Everything flows through Convex (a hosted realtime database with serverless functions). Content enters through scrapers and AI pipelines. Editors change it in /dev. The public site reads it. The colors below keep the same meaning for the rest of this page.
convex/*.ts) that read and write it. The Next.js app, the /dev editor, and the scripts all connect to the same Convex deployment. The public site reads through lib/convex/publicData.ts. The /dev editor writes through the /api/save-to-convex route. Scripts use an admin key.03 Tech stack — what each tool is for
App Router with server components by default: pages fetch data on the server and serve mostly-static HTML, prerendered at build time wherever possible. The app deploys on Vercel.
Hosted document database with TypeScript server functions (queries, mutations). The convex/ directory is the backend. Every environment — local dev, previews, production — targets the single production deployment (glad-minnow-656); the dev deployment is retired (see AGENTS.md).
auth.ts configures sign-in. middleware.ts protects /dev and, before launch, the production hosts (section 07). Editor roles (admin / editor / viewer) live in the Convex memberships table.
One API gateway to many models, selected by each workflow's config or entry module: Claude Opus 4.6 (summary, pharmacology), Claude Opus 4.5(harm potential, history & culture, legality, tolerance), Claude Haiku 4.5 (dosage & duration), Gemini 3 Flash (quote extraction), and DeepSeek V4 Pro (formal citations). Both the batch scripts and the in-editor section generator use it.
Utility-first styling with a token-based design system in src/theme/. Radix supplies accessible primitives. Framer Motion handles animation. React Hook Form drives the forms in the editor.
Bun is the package manager. Zod schemas (in src/schema/ and schemas/) validate every article shape before the app stores or renders it, so the app rejects malformed or AI-mangled content early. The repository has ~475 Vitest test files.
04 Repository map — where things live
Most of the code lives in four directories: src/ (the app), convex/ (the backend), scripts/ (the pipelines), and lib/ (server-side helpers shared between them). Everything else is documentation, assets, or configuration.
dose.wiki/ ├── src/ — the Next.js application (~900 files) │ ├── app/ — App Router: one folder per URL route │ ├── features/ — article, effects, reports, dev (~420 files) │ ├── components/ — shared UI: layout, pages, ui primitives │ ├── schema/ — Zod schemas validating article shapes │ ├── data/ — local JSON artifacts + generated maps │ └── theme/ — design tokens ├── convex/ — the backend: schema.ts + one file per table ├── scripts/ — data pipelines (~400 files; see section 08) │ ├── parsers/ batch/ citations/ prepopulate/ │ ├── convex/ migrate/ seed/ build/ analyze/ │ └── lib/ — shared CLI plumbing, OpenRouter client ├── lib/ — server-side helpers shared by app + scripts │ ├── convex/ — publicData.ts: how pages read Convex │ ├── auth/ http/ next/ citations/ runtime/ ├── auth.ts middleware.ts — sign-in config + /dev gatekeeper ├── public/ — molecule images, flags, favicons, SubstanceIndex.json ├── docs/ notes-and-plans/ — architecture docs, ADRs, plans ├── .agents/skills/legality-pi/— Pi legality research workflow └── README.md ARCHITECTURE.md CLAUDE.md CONTEXT.md
The repository keeps one local skill: .agents/skills/legality-pi/. Other automated workflows are documented beside their owning scripts under scripts/and docs/agents/, so executable behavior and operating guidance stay together.
05 Frontend — routes and feature modules
The code splits two ways. src/app/ holds thin route folders (one per URL, mostly data-fetching and metadata). src/features/ holds the rendering logic, grouped by domain. A route file fetches from Convex and passes the data to a feature module.
/[slug]— a substance article (the core page)/substances— browse all substances/effects,/effects/[effectSlug]— subjective effects/reports— trip reports/psychoactive · /chemical-classes · /mechanism · /category— classification indexes/interactions,/search,/about,/contributors/about#data— the Downloads section with three dataset JSON exports;/datais a compatibility redirect to/about/replications,/articles,/chemical-classes,/mantras,/blog,/documentation-style-guide— media gallery, articles, class pages, mandala, announcements, contributor style guide
/dev— editor home (auth required viamiddleware.ts)/dev/articles/[slug]— edit a substance/dev/change-log · tag-editor · index-layout · about · citation-review · profile/api/save-to-convex— the main write endpoint for the editor/sign-in— Auth.js sign-in entry
One codebase builds two publications — dose.wiki and the Effect Index sister site — selected at build time by NEXT_PUBLIC_SITE_FLAVOR. The route tree therefore also holds Effect Index surfaces; which ones render depends on the flavor (src/config/siteFlavor.ts).
article/— renders substance articles: dosage tables, duration charts, interaction grids, citation markerseffects/— effect pages, including a custom "VCode" markup renderer for effect descriptionsreports/— trip report pages
forms/— one React Hook Form section per article field grouptools/— substance editor + AI section generator, layout editor, citation review, about editorsave-orchestrator/— coordinates multi-endpoint savescontext/ · notices/ · profile/ · tags/ · prompts/— editor state, toasts, profile & tag editing
06 The Convex data layer — what is in the database
Each table has a matching convex/<table>.ts file that exports its queries and mutations. convex/schema.ts declares the shapes. There are three groups: content, editorial, and workflow. Readers see the content. The editorial records control who can change it and how. The workflow tables feed the AI pipelines.
| Table | Group | What it holds | Docs |
|---|---|---|---|
substanceIndex | content | The main table: one document per substance with every section — summary, dosage, duration, pharmacology, harm potential, legality, citations, references. | ~580 |
subjectiveEffects | content | Effect articles (for example "geometry" and "euphoria") with VCode markup bodies, galleries, and audio replications. | ~230 |
tripReports | content | Trip reports linked to substances and doses. | ~160 |
tripReportSubmissions | editorial | The private submission queue from the report form. Editors accept, reject, or promote a submission into tripReports before it goes public. | — |
replications / effectIndexArticles | content | Effect gallery media (video/image recreations of effects) and methodology articles. | — |
categoryLayout | content | The table that the public /substances page reads: the hand-curated psychoactive index layout. Saves to indexLayouts also update this table, so the page never reads indexLayouts directly. | — |
indexLayouts / siteConfig | content | Hand-curated classification trees (psychoactive / chemical / mechanism) and about-page configuration. | — |
moleculeOverrides | content | Hand-authored skeletal-diagram fixes from the /dev Molecules editor. When an override is present, the public article uses its SVG instead of the generated one. | — |
copyBlocks | content | Editable site copy, one document per block. The /dev Copy Studio edits it without a deploy. | — |
reagentTests | content | Point-in-time ProtestKit reagent-test results, keyed by substance slug. | — |
substanceGalleries | content | Per-substance curation of the Replication Showcase: a pinned head and a suppression list over replications slugs. | — |
memberships | editorial | Who can edit: email → role (admin / editor / viewer), synced from Auth.js sign-ins. | — |
changelog | editorial | Audit trail: when an editor saves, the table records who, when, and a markdown diff. Browsable at /dev/change-log. | — |
contributorProfiles | editorial | Public bios, aliases, links, and avatars for contributors. | — |
articleFeedback | editorial | Private per-article "report an issue / suggest an edit" intake. Editor-only; never rendered publicly. | — |
siteFeedback | editorial | Private site-wide general feedback intake. Editor-only; never rendered publicly. | — |
moleculeClassTemplates | editorial | Editor-authored plain-scaffold orientation templates for the Molecules editor. Editor-only cosmetics; saving one never changes a published molecule. | — |
articleSources | workflow | Raw scraped source text per substance, from the ten sites in the source list (section 08) — the input that the AI generators quote from. | — |
prompts / quotes | workflow | The editable AI prompts used for section generation, and quotes extracted from sources. | — |
citationEvidence | workflow | The citation audit trail: each claim in an article mapped to reference IDs, supporting quotes, and a review status. | — |
effectIndexArchive | workflow | Lossless Effect Index source records retained for a future Effect Index deployment. Not part of any dose.wiki read path. | — |
generatedPublicationOperations | workflow | Successful reviewed section publications. The proposal ID is the idempotency boundary for applying a generated section. | — |
07 Two surfaces, one database
Who: anyone, no login. The site is crawlable, SEO-tuned, and mostly static. (Before launch, the production hosts serve only the under-construction homepage at /. Previews and local dev serve the full site.)
How it reads: server components read Convex through lib/convex/publicData.ts (read-only URL). Flow A in section 09 traces this path step by step.
Key property: there is no client-side database access. The build can prerender pages. When an editor saves, the app revalidates them.
Who: authenticated editors only. middleware.ts redirects everyone else to /sign-in. The Convex memberships role decides what an editor can change.
How it writes: through the save-orchestrator and the /api/save-to-convex route. Flow B in section 09 traces this path step by step.
Source-backed generation runs only through authenticated local batch workflows. The /dev editor supports manual review and editing without exposing scraped source text in the browser.
There is a third consumer: anyone who wants the raw data. The /open-data/SubstanceIndex.json, /open-data/EffectIndex.json, and /open-data/TripReports.json routes serve daily Convex snapshots with per-dataset license envelopes. They are linked from the /about#data Downloads section; /data redirects to /about. Convex stays the live source for pages. The base data is CC0, but each dataset is "mixed" — the substance index keeps TripSit's terms on its interactionsfield, the effect index references third-party replication media, and legacy trip reports keep their authors' rights (see the license page).
08 Content pipelines — how the pipelines build an article
The pipelines build articles in stages. Parsers turn raw wiki text into structured data. Scripts load it into Convex and extend it with AI-generated prose. Editors review it in /dev. Then a citation workflow ties each claim to a reference. Each stage is a separate scripts/ subdirectory with its own npm run commands.
The scraped sources, in full. The parser registry (scripts/parsers/source-identity.ts) covers ten sites: PsychonautWiki, TripSit (three feeds: factsheets, wiki, and the combination guide), Erowid, Wikipedia, DrugBank, Isomer Design, SaferParty, The Drug Classroom, Disregard Everything I Say, and the Drug Users Bible. The corpus also stores a few further sources (Bluelight, D. M. Turner, Nervewing, ProtestKit) that have no parser.
The prompting system, in detail (quotes → prompts → sections)
The "AI section generation" box above is itself a two-stage process. The full scraped articles are too long and too mixed to give to a model directly. First, an extraction pass copies verbatim excerpts out of the sources, by topic. Then a stronger model writes one section from those excerpts, against a section-specific prompt that forbids claims not found in them. This section covers where the pieces live in the repository.
articleSourcesquotes table/devExtraction prompts (scripts/prompts/, scripts/batch/extract-quotes/) pull excerpts into eight topic buckets, defined in lib/quoteSections.mjs:
- Intro text · Harm potential · Pharmacology
- History & culture · Dosage & duration
- Tolerance · Legality · Subjective effects
Each article section has its own markdown prompt in src/data/config/sectionPrompts/— base, summary, dosage/duration, pharmacology, subjective effects, interactions, harm potential, tolerance, legality, history & culture, identification. They range from 4 KB (base) to 28 KB (pharmacology).
The repository files are seeds. The live copies sit in the Convex prompts table, and editors can change them in the /dev prompt workspace (registry: src/data/config/promptRegistry.ts).
How dose.wiki articles are made publishes every prompt unedited — the eight extraction prompts and the section prompts. It also documents the block-selection contract that keeps extraction verbatim, and the audit that enforces it. That page is the canonical copy of the pipeline instructions.
The citation workflow, in detail (scripts/citations/)
[cite:ref-id] markers only — article text must stay byte-identical otherwise--write --confirm-citation-writecitationEvidence[cite:reference-id] markers to the six citable sections, but must not change any other byte of article text. The tooling can use Wikipedia and PsychonautWiki to find references. It never cites them as claim support. CLAUDE.md records both rules, and the validation gate enforces them.09 Two flows worth tracing
Flow A — a reader opens dose.wiki/lsd
src/app/[slug]/page.tsx server component runslib/convex/publicData.ts queries substanceIndex.getBySlugsrc/schema/src/features/article/ sections, tables, citationsFlow B — an editor saves an article in /dev
src/features/dev/forms/save-orchestrator/ collects dirty articles/api/save-to-convex: auth role + rate limit + Zod contract validationsubstanceIndex.saveSubstance + auto changelog entry10 Recurring vocabulary
- Substance article
- The central data object: one large structured document per drug, with ~15 sections (dosage, duration, pharmacology, legality, and more). Defined by Zod schemas in
src/schema/, stored insubstanceIndex. - Citable sections
- The six prose sections allowed to carry citations:
summary,pharmacology,tolerance,harm_potential,history_culture,legality. Everything else (for example, dosage tables) is structured data, not cited prose. - Workbench
- A staging area for external review of citation drafts. The pipeline exports a task, and a person reviews the draft outside the repository. Then
citations:apply-workbench -- --write --confirm-citation-writewrites the approved result back to Convex — both flags are required. - VCode
- A custom markup format used for subjective-effect article bodies, stored alongside a parsed AST and rendered by
src/features/effects/. - Prepopulate vs. generate
- Prepopulate = deterministic import of parsed source data (high-confidence facts like dosage tables). Generate = AI-generated prose sections that a person always reviews.
- Convex deployment
- One production deployment (
glad-minnow-656). All environments — local dev, previews, production — read and write it directly; the old dev deployment is retired. Treat every Convex write as a production write (seeAGENTS.md).
11 Where to start reading
A reasonable reading order for a first session:
README.mdandARCHITECTURE.md— the orientation docs from the maintainers. They cover routes, npm scripts, and system layout.convex/schema.ts— the data model that everything else is built around. Skim thesubstanceIndextable definition first.src/app/[slug]/page.tsx→lib/convex/publicData.ts→src/features/article/— Flow A from section 09. This flow covers the whole public read path.middleware.ts+auth.ts— short files that show how/devis protected and how roles work.src/features/dev/save-orchestrator/andsrc/app/api/save-to-convex/— Flow B. These files show how edits reach the database with an audit trail.scripts/lib/workflow-command-surface.mjs— the registry of pipeline commands. From here you can find whichever pipeline (batch, citations, migrate, sync) you need.
Caveats about this page
Counts (substances, files, tests) are point-in-time snapshots and will drift — prefer the live tree over the printed number (e.g. find src -name '*.ts' | wc -l). This page also simplifies operational detail — the full citation validation rules, rate-limit policies. AGENTS.md, CLAUDE.md, CONTEXT.md, and docs/architecture/ in the repository are the authoritative versions.
Do you want the content side? This page covers the codebase. To see the full pipeline — scraping, quote extraction, AI synthesis, citations, and human review — read How dose.wiki articles are made.