Sizing and scale

db.md scales to millions of files natively: no embeddings index, no vector store, no external catalog. The directory is the database; the filesystem, embedded ripgrep, and a write-through catalog are the engine. The numbers below say how much data that actually covers, where the ceilings are, and when the packed-engine roadmap starts to matter.

These volumes are grounded in email, because a single user indexing their whole inbox is the canonical "a lot of sources" case. They are order-of-magnitude, with the assumptions stated so you can reproduce them.

Why the loop stays flat

One rule makes the scale claim hold: the interactive loop is designed around O(changed), not O(store). The operations the agent runs while it works (search, frontmatter lookup, backlinks, the pre-write dedup checks, the per-write catalog update, the post-write validate) should cost in proportion to what changed, not to how large the store is. Whole-store passes exist (dbmd validate --all, a full dbmd index rebuild, dbmd stats), but they are repair and audit operations, off the interactive path.

Four properties deliver it:

• Sources and event-type records date-shard; entity records and conclusions stay flat. Raw evidence never changes after it lands, so the toolkit parses each source once. When the agent writes through dbmd write, high-volume source and event types auto-partition by date (sources/emails/2026/05/, records/expenses/2026/05/) so no single directory holds an unbounded count and only the current shard is ever hot.
• Structured reads hit the index.jsonl sidecar; full-text reads are ripgrep. Attribute queries and dedup checks read one complete, sequential sidecar file per type-folder. Ad-hoc body search is embedded ripgrep. Never a full-store parse.
• The catalog is maintained write-through. Each write updates the human index.md and the machine index.jsonl for the affected type-folder. No full-store rebuild step in the loop.
• The log rotates. log.md is the active timeline; older months roll into log/<YYYY-MM>.md. The active log stays small regardless of store age.

Assumptions

• Volume. An average knowledge worker handles about 120 emails a day sent plus received (Radicati puts it at 121 to 126; Microsoft's 2025 Work Trend report puts received mail near 117). A heavy individual or marketer runs about 200 a day (sales send 60-plus, the C-suite receive 150-plus).
• Horizon. 365 days a year, indexing all mail including newsletters and automated messages. Working-days-only is roughly 30 percent lower.
• Bytes per email. The industry average is about 75 KB text-only (Mailmeteor). db.md has two cases: a text store at about 25 KB per email (HTML converted to markdown plus frontmatter, attachments referenced) and a full store at about 100 KB per email all-in (a 15 GB Gmail archive is roughly 100k to 150k emails, so about 100 to 150 KB each with attachments).
• Files. One email is one source file. Entity records (contact, company) add only a few percent, because they are deduped to the real-world entity set. Event records (expense, invoice, meeting, order, ticket) track business volume; for a company they can rival or exceed the source count, which is why event-type records date-shard like sources. The "few percent" holds for an individual, not a company.

Single user

File count is the metric that drives the tiers. Storage is shown as text-store / full-store.

Reality check: the file count, not raw bytes, is the thing to watch. A plain-text store stays small; attachment-heavy archives move the storage number first and the file-count number second.

Company

Naive sizing indexes every seat's mailbox. A blended seat is about 44k emails a year; the table is a five-year store.

Internal email is counted once per sender and once per recipient, so a naive per-seat sum double-counts; deduping unique messages roughly halves it. A real shared store indexes the relevant correspondence plus records and conclusions, not every internal CC. The result is smaller, but the order of magnitude holds.

What the split means

• An individual stays in the native sweet spot (50k to 1M files) for about a whole career. Plain files plus embedded ripgrep, fully git usable, is the right tool. Only a heavy marketer's 20-year archive (around 1.5M files) reaches the point where a packed on-disk format starts to pay.
• A shared store for even ten people can cross 1M files within two to three years. Mid-size teams hit tens of millions; large organizations reach billions and need a packed engine plus multi-store sharding.
• So the individual validates the simple toolkit, and larger shared stores force the engine. Company email alone justifies the roadmap, but company memory is a scale case, not the definition of db.md.

What actually binds

The volumes only matter against what the substrate can hold. Two facts dominate: a filesystem's theoretical maximum is far higher than the point where it gets slow, and the store and git-over-the-store have different ceilings. They are not the same limit.

Per-OS file ceilings

The max-files number is never the real constraint. Per-file stat() and open() cost on whole-tree operations is.

Two limits actually bind on every OS:

1. A flat directory tops out around 10k files. Past that, a cold ls, readdir, or sort degrades on ext4, APFS, and NTFS alike. Sharding fixes this: sources and event records date-shard, so no directory holds an unbounded count.
2. The whole-tree-walk wall starts to matter from hundreds of thousands of files. git status, ripgrep, and backups walk the entire tree; the cold cost is per-file lstat. Sharding does not help here, because it fixes per-directory lookup, not whole-tree walks. The write-through catalog keeps the agent's normal loop off this wall.

Git's own ceiling

Git's index is a single rewritten structure with one entry per tracked path, so status, add, commit, and checkout are O(all tracked files), not O(changed). This is the tighter limit, and it is an individual or small-team property, not the company story.

Git LFS does not apply here. It is for large binaries; db.md's text files delta-compress and merge.

The separation that matters: the filesystem plus ripgrep store reaches millions (sources and event records sharded), while git over the raw store caps near 1M. Very large history is the packed on-disk format and, on managed boxes, VM snapshots, which become the real audit log at that size. "Git is the audit log" is a persona-scoped claim, not a universal one.

The full spec

The headline volumes live in github.com/carloslfu/db.md/blob/main/SPEC.md under ## Scale, with the performance-budget table for loop and sweep operations. dbmd spec prints the same canonical text from the installed binary.