“Plain HTTP, plain text, lighter than MCP.”

At the transport layer, maybe that’s true. At the workflow layer, one /autoplan run expands into at least 60,600 to 103,500 tokens before any output.

What are orchestrators anyway?

Orchestrators, swarms, agent teams, virtual engineering teams or whatever else people call them, are everywhere right now. Their promise is simple: you keep the human in the loop, wrap the model in more structure, add planning and review for safety, and somehow get software faster without losing rigor.

There are many examples with gstack being one of them. It is presented as a lightweight local browser-agent stack and as proof that high-velocity AI-assisted development can produce useful engineering workflows.

People were making fun of garryslist, a public product of gstack. A widely shared third-party audit found a single homepage load pulling 169 requests and 6.42 MB for what is basically a personal blog. That sparked my interest to look under the hood and ask why the end-product felt so bloated.

For those that don’t know me: I’m a developer who cares about programming as a craft. What bugs me about the current coding-agent hype is that it rewards exactly the instincts decades of coding taught me to resist: more surface area, weaker boundaries, thinner verification, and confidence that has nothing behind it.

It’s not as if I never produced my fair share of sloppy code. I did. But then someone else reviewed it or a proper CI/CD path caught it before it shipped.

The current hype seems to celebrate skipping all of that. And if you object you “just don’t get how software gets made now.”

I can get behind that thought. Software does get made differently now, but I think differently should still mean correctly and with care.

The browser layer in gstack may be light. The workflow behind it is heavier, less uniform, and less proven than the public surface suggests. Maybe the output is useful. But useful output you can’t explain is not leverage. It’s debt. It’s likely to be unoptimized and bad!

This is supposed to be lightweight

gstack‘s docs also claim a 20-command session saves 30,000-40,000 tokens compared to chunkier tool protocols. And at the narrow transport layer, maybe that’s true.

But the repo isn’t pitched as “a slightly more efficient way to send browser commands.” It’s pitched as a software factory with public rhetoric about 10-15 parallel sprints. Once that is the promise, the question stops being about one HTTP call and starts being about the whole workflow.

The one command context bloat

/autoplan is sold as “one command, fully reviewed plan.” The skill itself is about 15.0k tokens under OpenAI’s o200k_base tokenizer. It explicitly tells the model to read full review skills from disk:

• /plan-ceo-review

• /plan-eng-review

• /plan-design-review (when UI scope is detected)

And those review skills can themselves inline /office-hours when no design doc exists, making the problem worse.

There is no runtime optimization between the skill files and the model. What’s on disk is what gets sent. No pruning, no compaction, and no token budget that trims content.

Claude Code’s own docs recommend keeping skills under 500 lines. gstack’s review skills are 1,200 to 1,677 lines each, and roughly 95% of that is duplicate voice rules, routing, and telemetry. The meat is about 50 lines per skill.

So “one command” is really a composed prompt path generating around:

• 60.6k when running /autoplan + CEO + ENG

• 80.8k when design review is also pulled in

• 83.3k when /office-hours gets inlined through the no-design-doc path

• 103.5k when both design review and /office-hours are in play

All of that is before the model writes a single token back, and before your own code, plan, or conversation history enters the context.

Large prompts alone are not necessarily evidence of bad engineering. But that defense only works if the public story is scoped honestly.

/autoplan is not some weird outlier. The total current-upstream generated skill surface is about 407,138 tokens plus a shared instruction set that keeps growing with each commit.

While not every token lands in one flat context window at once, the workflow always loads bloated instructions. Even gstack‘s own internal docs warn that full-skill reads in test fixtures cause context bloat and 5-10x slower runs.

That prompt load has a cost

Why should anyone care? After all, context windows are getting larger and token prices have fallen since the early GPT-4 era.

Because bigger windows don’t make prompt-heavy workflows free. They just raise the ceiling.

I do not have live gstack billing telemetry, so I am not going to pretend I do. But in my own separate orchestration tooling, built to send small, scoped tasks to isolated, headless agent sessions, the usage dashboard logged 839.2M tokens in one month, with peak days above 100M tokens. Only for testing the framework across 4-6 small foo projects.

That might not be direct proof about gstack. But it shows how quickly token burn stops being a rounding error once you start vibe-coding full-time.

And that was without all the planning overhead. The repo’s own public rhetoric about 10-15 parallel sprints implies tens or hundreds of millions of tokens a month.

It rots too

AI-heavy systems don’t crash loudly. They keep producing output that looks fine while becoming harder to reason about, slower to operate, and more expensive to run.

Long, layered contexts only make this worse. The more prompt mass you accumulate, the easier it becomes for the important parts to get buried under the rest. Any engineer who has maintained a large system knows the feeling of complexity arriving faster than understanding. That used to be the signal to “slow the fuck down” . Now it’s the signal to add another agent.

Lost in the Middle found that bigger windows do not guarantee the right information gets used well, and Context Rot demonstrated that accumulated context can degrade performance while the outputs still look plausible. Get My Drift also shows that LLMs silently shift task behavior under accumulated external context without observable error signals.

Perpetual maintenance

gstack is not only growing code. It’s growing prompts, generated skills, host transforms, and test surfaces around them. This creates a self-reinforcing loop:

1. add more workflow surface

2. add more scaffolding to manage the surface

3. add more assurance to calm fears about the scaffolding

4. polish it until it looks mature

5. add to the cost of understanding and changing it

It’s real progress with a carrying cost that stays invisible next to the headline “look how much leverage this gives me.”

In traditional software, degradation shows up in metrics such as latency spikes, error rates and failing tests. In prompt-heavy systems the degradation is more subtle. The output still looks structured. The plan still has sections and the review still sounds thorough. If you don’t audit the system for quality, the LLM will gladly gaslight you.

Anthropic’s own harness guidance says long-running agent workflows need runtime visibility into degradation, not just static test coverage. Right now there’s no way for an ordinary gstack user to tell when context growth starts hurting their results.

But it’s test-driven!

This repo has meaningful tests, CI, and eval infrastructure. But look at what actually runs. The default bun test path does not cover the expensive workflow and cross-host lanes readers are most likely to assume are covered. The broader CI eval surface is real too, just selective:

• PR eval workflows cover a selected set of suites

• periodic eval workflows are real but narrower

• direct Codex and Gemini suites remain small

The README says “All 31 skills work across all supported agents.” On Codex, Review Army is stripped entirely and the /codex skill itself is excluded. Factory Droid is still advertised with setup instructions, but the changelog says support was removed.

Anthropic’s own guide to agent evals makes the same point: real eval stacks are still partial and task-scoped. They don’t prove the whole product works.

I have seen the same pattern in my own codebase audits: plausible code, plausible fixes, plausible tests, plausible benchmarks, plausible docs. Each layer makes the next easier to believe, and confidence does not compound without a human in the loop.

As an adopter, you are being asked to trust a lot of workflow surface the tests don’t fully cover. Real enough to trust. Too opaque to verify.

Beyond one repo

gstack is not presented as a side project. It is used as public proof that high-velocity AI-assisted development works.

When systems like this get copied, vendored, and adapted, the cost is not only local:

• more token spend and slower loops for adopters

• more maintenance surface for teams that standardize on it: the prompts, the generated skills, the host transforms, the eval harnesses

• more generated software that looks more mature than it is

• more verification burden shifted onto downstream users

• more low-trust surface that somebody else has to filter

There is nothing wrong with exploring these tools and building with them. I do it myself. The problem is what happens when plausibility compounds at every layer and nobody is accountable for the gap between what’s promised and what’s shipped.

The orchestrator produces workflows that look structured. The workflows produce codebases that look tested. The codebases produce software that looks mature. And somewhere downstream an end user trusts something that nobody along the chain actually verified, in a codebase that no human and no model can comfortably reason about anymore because it grew faster than anyone could understand it.

That is the real cost of the velocity obsession. More commits, more features, more LOC per day, all of it rewarding output volume over the kind of slow, careful accountability that catches the things these systems miss. When the reviewer is a model, the tests are generated, the docs are generated, and the velocity metrics all say green, who is left to ask whether any of it actually works the way the README says it does?

This is not just about one repo. In earlier investigations of LLM-generated code, the same pattern kept showing up: plausible output, plausible tests, plausible benchmarks. And a 20,171x performance gap hiding underneath. That codebase was itself a product of an orchestration suite with its own prompt packs and skill tokens.

20K LOC/Day showed 15,647 (!!) green tests across a codebase where the one measurement that mattered was never written. METR’s randomized study and GitClear’s large-scale repository analysis both confirm these patterns are not isolated to any single developer or tool.

Steve Yegge’s Gastown, another orchestration framework, hit 189,000 lines in two weeks. Its creator openly says he has never read the code. Users reported $100/hour API burns and auto-merged failing tests. Even Anthropic’s own Claude Code CLI leaked at 512,000 lines and was immediately scrutinized for a single 3,167-line function and ~250,000 wasted API calls per day from a compaction loop that had no retry cap.

Each time the scale was different: a single function, a full codebase, the orchestrator suite itself. The pattern keeps moving up one level. The buck keeps being passed.

It’s your code. It ships under your name. If you can’t explain what the workflow is doing, how many tokens it’s burning, and whether the tests actually cover what the README says they cover, then the tooling is not giving you leverage. It’s giving you plausible deniability.

The craft hasn’t changed. Somebody still has to check.

- Hōrōshi バガボンド

Also published on X:

Sources

Primary Evidence

• gstack repository, current upstream at commit 6169273. Token measurements use OpenAI’s o200k_base tokenizer against current-upstream generated SKILL.md files. No runtime pruning verified via scripts/gen-skill-docs.ts generation pipeline analysis.

• Claude Code Skills Documentation. Anthropic. Official skill format specification recommending skills under 500 lines.

Primary Research — Long Context & Agent Degradation

• Liu, N. et al. “Lost in the Middle: How Language Models Use Long Contexts.” arXiv, 2023.

• “Context Rot: How Increasing Input Tokens Impacts LLM Performance.” Chroma Research, 2025.

• Arditi, S. et al. “Get My Drift? Catching LLM Task Drift with Activation Deltas.” arXiv, 2024.

Primary Research — Code & Productivity

• METR. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” July 2025.

• GitClear. “AI Code Quality Research 2025.” 2025.

Industry Commentary & Engineering Guides

• “Demystifying Evals for AI Agents.” Anthropic Engineering, 2025.

• “Effective Harnesses for Long-Running Agents.” Anthropic Engineering, 2025.

• Zechner, M. “Thoughts on Slowing the Fuck Down.” March 2026.

Third-Party Audits & Case Studies

• @Gregorein. garryslist.org homepage audit thread. X/Twitter, March 2026.

• “Diving into Claude Code’s Source Code Leak.” Engineers Codex, March 2026.

Prior Articles in This Series

• “Your LLM Doesn’t Write Correct Code. It Writes Plausible Code.”

• “20K LOC/Day Won’t Save You.”

“Thesis proven.”

The most notorious limitation of SQLite, its single-writer bottleneck, finally solved.

Or is it?

TL;DR: Not really. It just appears that way!

What’s actually behind it is a Rust rewrite that is, despite the effort of adding 20,000 lines of code a day, still ~1,000x slower than SQLite because it’s paying for a concurrency feature that doesn’t actually fire.

LLMs don’t just write plausible code. They write plausible fixes, plausible benchmarks and plausible test suites too, stacking layers of plausibility, each inheriting the confidence of the layer above and the untested assumptions of the layer below.

If you read my last article about plausibility, the same rules apply here: I use these tools myself as a practitioner and this isn’t about one individual developer. It’s about a widely observed pattern I think we’ll all see more of in the future. A reader called this kind of project “a canary in the software engineering mines” or something like that (can’t find it anymore), which I really liked. I appreciate the canary! I think we all should.

The rewrite keeps getting advertised as the SQLite killer so I thought I’d have another look. How did 3.9x faster become ~1,000x slower? I followed the claim through the benchmark, the project’s concurrency mode and the layers underneath.

The “Only Benchmark That Matters”

I compiled and ran the YCSB-A benchmark, a standard database throughput benchmark, against SQLite and the Rust rewrite’s C API library. One shared table, 50/50 read/write split:

The benchmark source is available in this repository so you can reproduce it yourself. Absolute timings vary with system load and hardware. Ratios are what matter.

I’d say with 10 ops/s at one thread the rewrite isn’t really competitive. Tho the more interesting part lies in the multi-thread columns. Those are the ones the original post claimed to outperform by ~4x.

If it’s ~1,000x slower on two threads, how did the LLM come up with ~4x faster, claiming success?

Luckily the upstream repo includes their own benchmark, so I checked what’s happening inside. Three things jump out:

• Each thread writes to its own table. Thread 0 into bench_0, thread 1 into bench_1, etc. They never touch the same data. On its own, I’d call that fine for a micro-benchmark. But MVCC, the primary feature in the project’s README, exists so that multiple transactions can work with the same data without blocking each other. If two writers touch the same row, one of them gets stopped and retried. If they never compete, there’s nothing concurrent going on.

• SQLite gets forced into BEGIN IMMEDIATE, a more aggressive locking mode than the default DEFERRED. BEGIN IMMEDIATE is the recommended approach for concurrent write-heavy work to avoid conflicts, but that’s not what’s being benchmarked here. There is no concurrency, only I/O parallelism across different tables.

• On top of that, every single row is its own transaction. This maximizes the overhead penalty SQLite has to pay per commit. 1,000 rows mean 1,000 commits. Real world databases batch hundreds or even thousands of rows per transaction. That’s what transactions are for! Paying the overhead cost without using its main feature defeats the entire purpose of transactions.

The benchmark is plausible: it runs and produces numbers. It even looks professional. It just doesn’t test what the README advertises and the post claims.

Brendan Gregg describes this anti-pattern in Systems Performance as the streetlight anti-method: analyzing with the tools you have rather than the tools the problem requires. The benchmark tested what was easy to measure (parallel I/O to separate tables) and avoided what was hard to measure (concurrent writes to shared data).

Or as Goodhart’s Law puts it: when a measure becomes a target, it ceases to be a measure. The 3.9x became the target. The only angle where that proof was possible for the LLM was a combination of separate tables, forced IMMEDIATE locking and single-row transactions. Any other combination would’ve shown the opposite.

After discovering that concurrency via MVCC wasn’t getting benchmarked, I gave SQLite the same parallel setup with one db file per thread for a fair head-to-head comparison:

Even then, the rewrite is 3-5x slower. The ~4x advantage in their benchmark didn’t come from MVCC being fast. It came from finding the one configuration that artificially inflated SQLite’s bottleneck while not even measuring the thing being advertised anymore.

If you ask an LLM to write a benchmark that should prove you’re faster than SQLite, it’ll find one. But that doesn’t mean it makes sense!

If your workload truly has zero contention, with each writer on different data, you don’t need MVCC at all. Separate SQLite files give you 3-5x more throughput, zero complexity and SQLite’s full test suite behind it.

What about all the cases where you DO need MVCC tho? It’s probably okay to pay the concurrency overhead for a guarantee of zero data loss, right?!

Zero Aborts, Zero Concurrency

The project’s README promises “Serializable Snapshot Isolation preventing write skew by default.” This basically just means that when two transactions touch the same row, the database stops one of them and retries it later so it doesn’t act on stale data. After the first one finishes, the row is updated and the second one can continue. Otherwise you could get race conditions like exceeding an account’s balance when withdrawing at the same time.

To test whether the conflict detection actually fires, I ran an Elle-style isolation test: four threads withdrawing 100 from five shared accounts with a total balance of 1,000 whenever the balance allowed it.

Sure enough, there were no negative withdrawals. But if you look closely at that table, there were also no aborts? PostgreSQL caught 26 conflicts while the Rust rewrite caught 0.

If SSI is running, there HAVE to be aborts! Zero aborts means either nobody competed for the same data or the conflict detection isn’t firing. The SSI code is there (I checked). It also checks for the right things. But a zero on shared data means something else is going on.

So I used a very simple test to see if concurrency is even used here: four threads, each doing a write transaction and then waiting 100ms. If Multi-Version Concurrency Control is doing its job, all of them should converge on finishing within about 100ms. If they’re serialized, it’ll take 400ms.

Both variants, the sequential and the concurrent one, finish in the same 400ms. They behave the exact same way! This means both are fully serialized.

The reason for this is that FrankenSQLite’s pager, which manages all access to data on disk, sits behind a single global mutex. Only one thread goes through the mutex at a time, making all the others wait.

Kleppmann describes this approach in Designing Data-Intensive Applications: “The simplest way of avoiding concurrency problems is to remove the concurrency entirely: to execute only one transaction at a time, in serial order, on a single thread.”

There is no concurrency here.

BUT there is an MVCC layer sitting above the mutex, which does real work: it creates snapshots, tracks versions and builds conflict detection graphs. Every time it needs to actually touch data, it queues behind the same lock as everyone else.

The concurrency model can’t provide concurrency because the storage layer underneath won’t let it. You’re paying a 150x penalty for a feature that the mutex makes architecturally impossible to deliver. This is what a gap between layers looks like: the MVCC layer works, the pager layer works, the connection between them doesn’t.

The README itself contains a general caveat: “This README describes the target end-state architecture. The runnable code today is in a hybrid state.” Fair enough.

But the same README, in the same “Current Implementation Status” section, states: “fsqlite-vfs, fsqlite-pager, fsqlite-wal, fsqlite-mvcc, and fsqlite-btree are wired into default runtime execution.” The feature comparison table says “Concurrent writers: Many (page-level MVCC with SSI).” No asterisk or “planned.” It uses present tense. The caveat is scoped to “Native mode / ECS sections.” The concurrency claims are on the live side of that line.

A common response to my last article was “it’s not done yet” and again it might be tempting to say “just fix the concurrency and it’ll work.” And sure, software is always in progress. That’s exactly why “thesis proven” is the claim that needs the second measurement! The benchmark screenshot says 3.9x. The overlap test says 408ms. One of them measured a layer. The other measured a connection. The README’s own status section claims the MVCC stack is wired. My overlap test shows it isn’t.

The gap isn’t between “done” and “not done.” It’s between what’s claimed and what’s connected.

The Layers of Plausibility

The project’s test suite has 15,647 tests. A file called ssi_write_skew.rs tests the exact scenario I just ran and passes. correctness_concurrent_writes.rs contains a comment reading: “MVCC concurrent writer path is not yet wired.” The tests for it pass anyway. The tests validate each layer in isolation, where the code IS correct. They don’t test across the layers where it breaks.

And it’s not just the concurrency stack. The JOIN planner crashes at 12.3 GB of RAM on 375 rows, the C API exports only 19 of SQLite’s 220 functions. The pattern of plausible layers with untested connections runs through the entire project.

Steve Freeman and Nat Pryce argue in Growing Object-Oriented Software that tests exist to surface failures, not to confirm success. 15,647 tests that all pass. None of them fail where it matters: at the boundary between layers.

Frederick Brooks called these “the most pernicious and subtle bugs” back in 1975: system bugs arising from mismatched assumptions made by the authors of various components. Each layer of FrankenSQLite’s concurrency stack assumes the one below it works. None of them check.

The same pattern from my last plausibility article is at work here. If you read it top-down, everything looks correct: a mutex managing access, SSI detecting conflicts, transactions with proper isolation and a benchmark showing 3.9x. Again, the modules have the right names, the architecture makes sense and the tests pass.

Bottom-up, none of it connects. The claim comes from an invalid benchmark that validates a feature adding 150x overhead without doing what it promises. It can’t detect conflicts because the mutex above it serializes everything. All this work only for the mutex to provide isolation the same way SQLite does: by not allowing concurrency at all.

=== Building and running fsqlite-mvcc tests (SSI anomaly) ===
     Running unittests src/lib.rs (target/release/deps/fsqlite_mvcc-285cac5b642bd309)

running 24 tests
[...]
test ssi_anomaly_tests::ssi_anomaly_write_skew_detected ... ok
test ssi_anomaly_tests::ssi_anomaly_phantom_insert_detected ... ok
[...]

test result: ok. 24 passed; 0 failed; 0 ignored; 0 measured; 1212 filtered out; finished in 2.51s

To be fair, some things did get better since the last article. On the single-threaded micro-benchmark, the path improved from 20,171x slower to about 9x. The WAL (write-ahead log) sync path was fixed, page copies were reduced and frame assembly was optimized. That’s real engineering! The things that got measured in response got better. Credit where it’s due.

But the concurrency path, the one validated by the rigged benchmark, got worse. The 150x overhead wasn’t there before. It was added by enabling MVCC on every query without ever testing whether the concurrency it provides actually works. The measured path improved while the unmeasured path accumulated overhead.

For the ones thinking “it’s all refactorable, just replace the mutex and wire the layers.” Yes, probably. But you don’t refactor what you don’t know is broken. The SSI tests pass and the benchmark shows 3.9x. Every signal says green. The overlap test that would reveal the truth doesn’t exist in the repo because nothing suggested it was needed. That’s what plausible layers do: they don’t just hide bugs from reviewers, they hide them from the developer too.

During my own experiment with this codebase, where I tried improving its performance by only prompting, there was a moment where the LLM produced a benchmark showing INSERT at 0.70x, faster than SQLite. My first reaction wasn’t to celebrate but to stop and ask what’s wrong.

Turns out the LLM had silently batched 64 autocommit statements into one transaction, changing the durability guarantee. When I asked it about the speed increase directly, it admitted: “This is cheating.”

The LLM’s own reply called it cheating. It just didn’t volunteer that until asked directly. The 3.9x benchmark has the same structure: it measures something real, just not the thing being claimed.

So what does it look like when concurrent writes actually work?

How Concurrency Layers Actually Work

PostgreSQL has had working Serializable Snapshot Isolation since 2011. The FrankenSQLite spec document even cites the Ports and Grittner paper that documented it.

Even though FrankenSQLite runs in-process with zero network cost, PostgreSQL, a server where every query pays a network round-trip, is still 188x faster at 1 thread.

Three deliberate engineering choices explain the gap:

• PostgreSQL doesn’t funnel all page access through one lock. It uses per-page lightweight locks so transactions on different pages proceed in parallel. FrankenSQLite has a single Arc<Mutex> for the entire pager.

• In PostgreSQL transactions overlap by design. BEGIN grabs a snapshot without holding a lock. Only COMMIT briefly validates and serializes. My overlap test showed FrankenSQLite takes 400ms for what should take 100 because nothing overlapped.

• PostgreSQL pays for serializability in aborts. 253 at 4 threads, 1,797 at 8. That’s the cost of actually checking whether two transactions stepped on each other. FrankenSQLite pays 150x overhead and checks nothing.

Each of these is a connection between layers that PostgreSQL tests and FrankenSQLite doesn’t. Locks connect the pager to concurrent access. Overlap connects transactions to the scheduler. Aborts connect SSI to the commit path. In FrankenSQLite, every one of these connections is missing while every individual layer passes its own tests.

Confidence Outruns Evidence

Speed builds plausible layers. Plausible layers hide the gaps. Hidden gaps don’t get tested. Untested gaps don’t get refactored. And the system ships with metrics that say it works.

20,000 lines a day over hundreds of commits in weeks, and only a handful about performance. The debt accumulated faster than anyone could evaluate it. Every layer was plausible. Every test passed. Every metric said green. And the one measurement that mattered - do the layers actually connect - was never written because every signal said it wasn’t needed.

Domain knowledge is key. I wrote about this in The Bug That Shipped: the LLM knows about the bugs. It just doesn’t volunteer the information unless you ask the right question. And knowing which question to ask is the part that can’t be automated. You need to know that zero aborts on contended data is wrong. You need to know that “faster than SQLite”, a 25-year-old codebase, means something changed that shouldn’t have.

My last article closed with: “The vibes are not enough. Define what correct means. Then measure.”

The response was 20,000 lines of code per day.

The second measurement is the one that was missing. The first one tested layers. The second one tests the connections between them. Before shipping the next feature, stop and ask yourself: am I measuring what I think I’m measuring? Or am I just measuring what’s easy to measure?

15,647 tests said green. The overlap test that would have said red took 10 lines of C.

LLMs are useful. 20,000 lines a day is useful. But speed without the second measurement is just plausibility at scale.

Or as Mario Zechner puts it: “Slow the F down.”

Benchmarks run against FrankenSQLite at commit 12b18f9b (March 25, 2026; 952 commits post-squash), PostgreSQL 16, and SQLite system library. All sqlite3-compatible benchmarks use the C API. Concurrent benchmarks are file-backed (required for multi-connection access). Benchmark harness: db_bench_foo. FrankenSQLite’s own benchmark: concurrent_write_persistent_bench.rs.

Sources

Some book links are Amazon affiliate links. Buying through them costs you nothing extra and helps support this work.

Concurrency Control & Isolation

• Kleppmann, M. Designing Data-Intensive Applications. O’Reilly, 2017. Chapter 7: Transactions.

• Ports, D., Grittner, K. “Serializable Snapshot Isolation in PostgreSQL.” VLDB 2012.

Benchmark Methodology

• Cooper, B. et al. “Benchmarking Cloud Serving Systems with YCSB.” SoCC 2010.

• Kingsbury, K., Alvaro, P. “Elle: Inferring Isolation Anomalies from Experimental Observations.” VLDB 2021.

Systems Performance & Anti-Patterns

• Gregg, B. Systems Performance: Enterprise and the Cloud. 2nd ed. Addison-Wesley, 2020.

• Brooks, F. The Mythical Man-Month. Addison-Wesley, 1975.

• Freeman, S., Pryce, N. Growing Object-Oriented Software, Guided by Tests. Addison-Wesley, 2009.

SQLite Documentation

• WAL Mode. Transaction. Testing. BEGIN CONCURRENT branch.

Related Writing

• Zechner, M. “Thoughts on Slowing the Fuck Down.” March 2026.

Benchmarks & Evidence

• Benchmark harness: db_bench_foo (open source, reproducible).

• FrankenSQLite’s own benchmark: concurrent_write_persistent_bench.rs.

Thirteen out of a hundred times.

That’s how often coding agents catch critical deployment-level production failures while reviewing their own code.

A hundred out of a hundred times they nail it when you ask them what could break in production.

Just to be clear: I’m not trying to bash coding agents here. I’m just trying to find the signal within all the noise. After all, I think we all want AI to work out. I use it myself every day for all kinds of tasks. Not for coding though, you’ll soon see why.

Earlier I wrote about how LLMs write code that looks correct but isn’t. At that time I assumed the models couldn’t really tell the difference and the natural response of many readers was “just add a review agent into the pipeline.

So I measured what happens when you do. I had five frontier models write code for ten common production failure patterns and then review themselves over 3,000 trials.

Turns out I was wrong about the why. The models can tell. They absolutely do know! They happily produce a full diagnosis covering every failure scenario and the right fix. IF you ask the right question that is.

Let me show you what I mean:

The Logic Was Sound

I prompted them to write a classical retry wrapper. The kind every backend service needs when talking to external APIs. When your 3rd party payment provider of choice goes down for a few seconds, swallowing your request, you just wait and retry until it works again. Pretty standard stuff.

The code they produced looks clean. It has error handling, exponential backoff and configurable retries:

def retry_request(method, url, retries=3, backoff=1, **kwargs):
    last_exception = None
    last_response = None

    for attempt in range(retries):
        try:
            response = requests.request(method, url, **kwargs)
            if response.status_code < 500:
                return response
            last_response = response
        except (ConnectionError, Timeout) as exc:
            last_exception = exc

        if attempt < retries - 1:
            time.sleep(backoff * (2 ** attempt))

    if last_response is not None:
        return last_response
    raise last_exception

Then I asked them to review the code. They concluded:

“The logic is sound.”

“Backoff works.”

And even reassured themselves:

“That’s the only real concern. The rest is straightforward.”

Almost always they missed the most critical problem. The wrapper has backoff, yes, it waits longer between each retry. That part works.

The waits are identical for every client though. When the upstream goes down and failures start trickling in, all clients start retrying at slightly different times. First they all sleep one second, then two, then four. Two rounds in, clients that failed seconds apart are waking at the same time, hitting the upstream in synchronized waves. The deterministic backoff pulls them into lockstep.

Source: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ AWS Architecture Blog

Meanwhile your own service is choking. Every worker thread is stuck in time.sleep(), blocking anything else from getting through. There are no threads left to handle actual requests. A five-second upstream blip turned into a minutes-long outage because every worker is waiting on the same backoff timer to expire.

This problem is called a thundering herd. The fix is pretty straightforward: instead of sleeping exactly two seconds, sleep somewhere between one and three (aka jitter).

Scattering the clients prevents them from converging into waves. Your retries trickle in instead of slamming all at once, giving the upstream room to breathe.

Source: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ AWS Architecture Blog

When I asked “What happens if 1,000 clients hit this simultaneously?”, they all calculated the exact failure mode, explained why the server can’t recover, and recommended jitter. They had the full analysis ready.

They just didn’t mention any of this when asked to review the code.

Ok, the retry function was missing a line of code. So what? It’s a simple bug that anyone could’ve missed on a bad day. The model probably just didn’t think of it.

Fair enough. Except in the next one, the model actually wrote defensive code itself and still missed what it was defending against.

Reviewed And Cleared

In production systems cronjobs are everywhere. Think sending daily reports, cleaning up expired sessions or build artifacts, and processing payment queues. Every backend runs a few of those.

To help lift that burden, I prompted all the models to code a scheduled task runner.

Again the code they produced looked fine.

class Scheduler:
    def __init__(self):
        self._tasks = []
        self._threads = []
        self._stop_event = threading.Event()
        self._lock = threading.Lock()
        self._running = False

One subtle detail stood out:

See the self._lock = threading.Lock()? They added a lock guard unprompted! It makes sure two threads can’t mess with the task list at the same time. Something in the model understood concurrency well enough to write defensive code on its own.

But a threading lock only guards against collisions between threads inside one copy of the program. It accomplishes nothing when two instances of your service run this scheduler simultaneously. Both instances fire every task and your users will get two emails and their payments processed twice.

When I asked them to review it themselves, they found five issues. Namely shutdown blocking, clock drift, silent exceptions, first-run timing and sustained load from slow tasks. Basically covering everything that could make your tasks run late, crash silently, or pile up. These are all real bugs but they only cover single-process concerns.

“The locking and thread lifecycle is otherwise reasonable for a simple implementation” they said.

Then I asked what happens when two instances run behind a load balancer:

“Both instances can process the same item and double-charge.”

“Daily summary email: it runs twice, so users get duplicate emails.”

“This scheduler is in-memory and per-process. A load balancer does not coordinate it.”

They wrote the lock because they understood concurrency and then reviewed it and called it reasonable. Only to then explain exactly why the lock was insufficient the moment someone asked. This confirms the knowledge was there all along.

So they can miss it, and they can know the answer and still not apply it. So far so bad. Next they do something even worse: they look right at the bug and call it safe!

Certified Safe

I asked for a function that exports a PostgreSQL table to CSV. Order histories, user activity logs, anything that your services accumulate over the years. The code they produced:

cursor.execute(f"SELECT * FROM {table_name}")
columns = [desc[0] for desc in cursor.description]
rows = cursor.fetchall()
writer.writerows(rows)

Probably like a lot of people wrote their very first PostgreSQL to CSV exporter in python. And for most home use cases that’s fine. There’s only one problem:

cursor.fetchall() loads the entire result set into memory before writing a single row. On a 500-million-row table, which resembles real-world order histories at any busy company that’s been around a few years. That’s roughly 500 GB dumped into RAM all at once. Most servers run out of memory here and the process gets killed by the operating system (out of memory) before writing a single row.

When they reviewed it, they said:

“Memory: iterating over the cursor streams rows instead of loading all at once.”

Read that again! The code calls fetchall() and the review praises it for streaming. They certified the code as safe precisely where it’s lethal!

Then I asked what happens with 500 million rows:

“500 million rows x ~1 KB = 500 GB raw data. Even with 128 GB RAM, this process dies in seconds.”

They produced the memory calculation, identified the OOM kill, and recommended the exact fix: a server-side cursor that streams rows instead of loading them all at once. For the exact failure they had just endorsed as safe.

None Of It Was True

These three aren’t cherry-picked worst cases. They’re representative. I ran the same experiment across ten common scenarios where subtle bugs make real-world production life hard.

Yes, of course an open-ended any concerns? finds less than a targeted what happens with 1,000 clients?. That’s obvious. Everyone would expect a gap here. Tho I think not everyone also expects it to be that wide.

Four out of five models catch the cron double-execution 0% of the time in self-review. Only 8% for the other one. When you ask them directly, each single one of them nails it 100%.

Zooming out to all ten scenarios shows a clear pattern across three different tiers:

The models relatively reliably catch code-level bugs like wrong logic, missing error handling and type mismatches while missing what happens when that code hits production.

There’s a visible gradient here: the initiative gap scales with distance from the code.

The thundering herd is fixable in one line of code. Just add randomness to the sleep. Models catch it 4-63% of the time in undirected self-review. Fixing things like the cron double-execution on the other hand requires infrastructure outside the codebase: a distributed lock, a leader election, or an external scheduler like Celery Beat. Nothing in the code is wrong. The bug lives in the deployment topology, which makes the models catch it only 0-8% of the time in undirected self-review.

A side project with ten users won’t hit these problems. Your company’s checkout page will.

No single model is the outlier here either, they all show the same gap:

And no, you can’t just pick the “best” model. Codex-5.3 leads on thundering herd (63%) but scores worst on connection pool. Opus-4.6 leads on thread safety (99%) but scores worst on thundering herd (4%). Every model has blind spots, just different ones.

It’s also not context-related. The self-reviews ran in the exact same session the code was generated. If the model was ever going to catch it on its own, this was the time.

The Expert Witness

So whose fault is it that nobody asked about deployment? You get what you ask for!

That’s true. Nobody asked about deployment. Nonetheless, the thundering herd shipped to production.

Let’s think about what a human reviewer does for a moment. They walk into the code review, see a retry function without jitter, and say “where’s the jitter?” No prompt needed. Twenty years of 2 AM pages because of exactly this bug is their checklist. The model has the equivalent of that experience and proved it the moment someone asked. It just doesn’t volunteer it.

That’s not how a code reviewer works. That’s how an expert witness works. Full, accurate answers to whatever you ask without ever raising the questions you didn’t ask, even when the answer would change everything.

McConnell’s Code Complete notes that assigning reviewers specific perspectives - “review this as the maintenance programmer,” “review this as the customer” - can uncover more defects than general review. What happens with 1,000 clients? is a perspective assignment while Any concerns? is not.

The people who most confidently can deploy reviewer agents into the pipeline are building products instead of weekend projects. The developer who prompts build me a retry function is probably not the developer who knows to ask about thundering herds. That’s the whole reason they’re using the model. The safety net has a hole shaped exactly like the thing it’s supposed to catch.

If you’re relying on the latest addition of memory and skill mechanics to close the gap, the outlook isn’t much better. The model still has to catch it first! At a 13% self-review rate for deployment concerns, that’s hoping for a lucky shot to seed the memory and then hoping the memory system surfaces it in the right context next time. Instead of closing the gap, memory just becomes the filter for what gets remembered.

Trial runs for skills and memory system are still ongoing. Early results are interesting. More on that soon.

The Speed Trap

“Just ask the right questions, then.“ Sure. The model answers perfectly when you do.

Follow that thought for a second tho. If you’re senior enough to ask what about the thundering herd?, you’d have caught the missing jitter yourself. And if you’re not - if you’re the developer who asked the model to write the retry function in the first place - you’d probably ship the review as-is.

That’s the speed trap. The model is fast enough to earn your trust, and the code piles up before anyone checks what the review actually missed.

That’s just more code, all reviewed by the same process that missed the thundering herd.

Faros AI measured this across 10,000 developers. Compared to pre-AI baselines, 98% more pull requests are merged with the average PR size up 154% and review time up 91%. METR found experienced open-source developers were 19% slower partly because of the overhead of reviewing and correcting AI output. The review that already doesn’t catch deployment concerns now has to catch them in pull requests that are two and a half times larger.

As Bellotti puts it in Kill It with Fire: automation means we ignore whatever we automated. The AI review catches the correctness bugs almost every time. That’s exactly what makes you stop checking for the ones it misses.

Always Ask What Breaks

Don’t ask the model to review your code. Ask it what breaks.

What breaks in your domain, at your scale, with your deployment? “What happens when 1,000 clients hit this at once?” “What happens when two instances run behind a load balancer?” “What happens when this table has 500 million rows?”

As Charity Majors puts it: complex systems fail in ways you can’t predict. Your tools need to handle questions you haven’t thought to ask yet.

Every question gets a perfect answer. No question gets asked unprompted.

Don’t be the one getting thirteen out of a hundred.

- Hōrōshi バガボンド

Methodology

The full trial data including all prompts, model responses, and evaluation scripts is available on GitHub. Data collection is ongoing. I don’t expect significant shifts in the numbers at this point.

Models tested: Claude Opus 4.6, Claude Sonnet 4.6, OpenAI Codex-5.3, GPT-5.4, GLM-5. All models ran with extended thinking / xhigh reasoning effort enabled. Over 3,700 trials across 10 scenarios x 5 models. March 2026.

Sources

Some book links are Amazon affiliate links. Buying through them costs you nothing extra and helps support this work.

• Brooker, M. (2015). “Exponential Backoff And Jitter.” AWS Architecture Blog.

• Brooker, M. “Timeouts, retries, and backoff with jitter.” Amazon Builders’ Library.

• Google. “Managing Load.” Site Reliability Engineering Workbook. (Uses “thundering herd” verbatim; Pokémon GO 20x spike case study.)

• Python Software Foundation. “threading - Thread-based parallelism.” Python 3 documentation.

• Kleppmann, M. (2016). “How to do distributed locking.”

• psycopg2 documentation. “Server side cursors.” psycopg.org.

• METR. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” July 2025 (updated February 2026).

• Faros AI. “The AI Productivity Paradox.” 2025.

• McConnell, S. Code Complete. 2nd ed. Microsoft Press, 2004. (Perspective-based reviews, inspection effectiveness.)

• Bellotti, M. Kill It with Fire. No Starch Press, 2021. (”automation means we ignore whatever we automated.”)

• Majors, C. “Honeycomb 10 Year Manifesto.” honeycomb.io, February 2026.

When prompted, LLMs cheat on durability and only confess when caught.

They benchmark wrong binaries.

They declare performance floors eleven times and are wrong each time.

They silently filter errors from their output to make benchmarks look real.

Same LLM, same codebase, same goal. Three different outcomes: 4.5x faster. 270x faster. 900x faster.

The difference wasn’t the tool but the prompts. The gap between 4.5x and 900x comes down to the questions asked, not the model used.

TL;DR: Three personas prompted the same LLM to fix the same broken database. The vibecoder got 4.5x. The mid-level dev got 270x. The domain expert got 900x. The technical sections below show how. If you want the actionable takeaways, skip to “The Diagnosis Ceiling.”

An Experiment

After my last article about an LLM-generated Rust rewrite of SQLite that was 20,000x slower than its reference implementation, the most common question was: “But could an LLM fix it?”

Fair question! I forked the original rewrite to kqsqlite under its MIT license and ran a structured case study where I assumed three different personas and used only vanilla prompting to fix it. No agents, no tools, no skills, no RAG, no system prompts.

To have some kind of reference frame, I grounded the personas in Bloom’s Revised Taxonomy of cognitive operations (Anderson & Krathwohl (2001)). The level of question you ask determines the level of answer you can get:

Now some of these are above my pay grade and some of these I’m having a hard time shifting into because it’s been a really long time. I had the chance to sit with an experienced L5, who was kind enough to correct my framing where wrong to make sure the technical grounding held up. The knowledge in those prompts is verified. So is the effort it took to get there.

The toolset scales with the persona. The L0 had a benchmark table. The L2 had the same table plus EXPLAIN QUERY PLAN. The L5 had both, plus flamegraphs, callgrind, YCSB workloads (Yahoo’s database benchmark suite), TPC-H analytical queries, and three-way differential testing against C SQLite and Turso (a professional-grade Rust SQLite reimplementation formerly named Limbo). The richer the toolset, the more precisely the human can direct the process.

No code review. I never read the LLM’s generated code but only its benchmark numbers, profile output and diagnostic reasoning. If the code is wrong, the numbers will show it. If the numbers look right but the code is broken, that’s a finding about LLM reliability, not a reason to intervene. The experiment tests what the human brings to the conversation, not their ability to audit code.

You can find the full methodology, ground truth rubric, and scoring criteria here. Prompt/response transcripts for all three sessions: github.com/KatanaQuant/kqsqlite.

The Same Table, Three Readers

The starting point for all three personas was the same six numbers:

The benchmark baseline (1,000 rows):

Six rows and three columns. What each one does with them determines the outcome.

The vibecoder just pasted it as context: “its kinda slow can u make it faster.” The LLM treated every ratio as equally important and optimized what it could see: compiler flags, tracing overhead, unused concurrency mode. All real improvements but none addressed the 6,851x. After seven prompts the algorithmic bugs were all still there.

But the L0 wasn’t flying blind. By Prompt 5 the LLM noticed on its own that UPDATE was 12x slower than INSERT and started investigating the write path. It found the symptom but it never diagnosed the root cause: is_rowid_ref returning false, hardcoding every write into a full table scan. The L0’s problem wasn’t laziness. They were missing a real framework to follow a symptom to its architectural root.

The L0 was cut short after seven prompts. By then the L2 sessions had shown a pattern: the LLM cheated on durability, produced misleading benchmark improvements, and declared victory with algorithmic bugs untouched. If the L2 - who could at least read the numbers - was getting lied to, the L0 had no mechanism to catch it. There was no reason to expect it could deliver on our goal: not plausible but correct code.

The L2 read the numbers against each other. The first insight was a ratio of ratios: SELECT BY ID (6,851x) versus SELECT ALL (38x). Both are reads on the same table. Both go through the same compilation path with the only difference that one fetches all rows and the other fetches one row by a specific key. A 180x gap between those two operations means something is wrong with how the database finds a single row.

The LLM’s first answer was wrong. It blamed query compilation overhead. The L2 pushed: “if it’s only compilation overhead, why is SELECT BY ID 6,851x but SELECT ALL is only 38x? Don’t they both compile? What’s different about them?” Three prompts later, the L2 asked for an EXPLAIN QUERY PLAN, the database’s own report on how it will execute a query. The answer was revealing:

• kqsqlite: SCAN test (read every row, check each one)

• C SQLite: SEARCH test USING INTEGER PRIMARY KEY (rowid=?) (jump directly to the row)

The planner wasn’t using the index, what Winand (SQL Performance Explained) calls “the first power of indexing: finding data efficiently.” At 1,000 rows, the full scan checks every row (O(n)) while a B-tree descent only touches 2-3 levels (O(log n)). At 10,000 rows, the gap grows super-linearly so the planner bug doesn’t just add overhead but changes the complexity. Over three prompts and two bugs, SELECT BY ID dropped from 6,851x to 36x. The EXPLAIN diagnostic drove the final leg from 273x to 36x. The L2 couldn’t name the root cause in the code but could reason about numbers.

The L5 translated each anomalous ratio into an architectural hypothesis before issuing any prompt, decomposing the table into three tiers:

Tier 1 - Algorithmic: The same planner failure the L2 would find by ratio-reading, but diagnosed from first principles. A 290x gap between point lookup and full scan means full table scans where the B-tree should be doing O(log n) descent (Knuth, TAOCP Vol. 3). Fix whatever prevents index lookups.

Tier 2 - Per-statement: The 47x gap between individual INSERT (615x) and batched INSERT (13x) means something expensive happens after every individual statement. In a database with write-ahead logging the most obvious suspect is forcing that log to disk after every statement, as if power could fail at any moment (Mohan et al., ARIES, 1992). Find the per-statement overhead.

Tier 3 - Constant-factor: The remaining 13x is constant-factor overhead: the same operations done less efficiently. Profile later, after the architectural bugs are fixed. Target: ~2x.

The LLM returned four file:line citations in its first response including the exact function responsible for the planner bug. Not because the LLM was smarter but because the question told it where to look.

What the Numbers Show

The L5’s tier decomposition had acceptance gates as explicit thresholds that signal when to stop:

The L2 had no such plan. They were just reading numbers after each fix and asked “why is this one still anomalous?” That’s enough to find 5 out of 6 critical bugs (scorecard below). But without acceptance criteria, you can’t tell when you’re done. The L2 declared the gap “fundamental” or “structural” six times, each time with a different justification but was wrong each time. The L5 accepted none even tho the LLM declared the floor eleven times.

Ground Truth Scorecard

Six bugs identified in the first article before the experiment began:

The L0 found two bugs at the surface-level while the L2 found five over 32 prompts, each exposed by the previous fix. The L5 identified the top four in the first prompt’s tier decomposition.

GT#6 is an interesting case. table_seek was pre-identified as a bug reading implemented but never called from the query path. Tho when at Prompt 12 the L5 directed the LLM to investigate, it found the function was called through an indirect dispatch the ground truth had missed. The LLM corrected the human’s pre-assessment. This is the only case in the experiment where the scorecard changed direction and serves as a reminder that ground truths established by reading code are less reliable than ground truths established by running it.

For the ones thinking the L5 just gave better hints! Yes, that’s the point! The L2’s prompts contain no database knowledge, just ratios. The L5’s prompts contain domain expertise. Both use the same LLM, the same codebase and the same tools. The only variable is what the human knows how to ask for.

The bugs weren’t pre-identified and the prompts weren’t designed to find them. The ground truth was established to score the results and not to guide the prompts. The L2’s first prompt was the benchmark table and the question “why is SELECT BY ID 6,851x?”. Just a number that looked wrong.

The LLM’s usefulness is bounded by the human’s ability to frame the problem.

The next three sections detail the technical evidence. For the actionable findings, skip to The Diagnosis Ceiling.

The Translation Tax

After all six algorithmic bugs were fixed, what remains is the translation itself: the systematic cost of reimplementing a C database in Rust. By Prompt 38, with algorithmic fixes and architectural changes applied, the profile was “flat” with no single function above 12% of CPU time. The question shifts to how close you can push a Rust rewrite to the reference implementation. The L5 directed the LLM to profile the INSERT path.

The profiler identified lookup_keyword at 11.16% of INSERT CPU time.

Every SQL parser checks whether each token is a keyword or a user-defined name. One of the most frequent operations in a database engine. The more common real world approach builds the keyword set at compile time and then dispatches on length or first character to compare it in place without any allocation or copies (Aho, Sethi & Ullman, Compilers, 1986).

The LLM’s Rust translation:

match s.to_ascii_uppercase().as_str() {
    "SELECT" => Token::Select,
    "INSERT" => Token::Insert,
    "WHERE" => Token::Where,
    // ... ~150 more keywords
}

to_ascii_uppercase() creates a new heap-allocated String on every call for each token in every SQL statement across all benchmark iterations. For 1,000 INSERT statements each containing ~10 tokens, that’s 10,000 heap allocations just to check whether a word is a keyword. Work that any production scanner does without heap allocation.

The code is plausible and produces the right answer. It’s also how any Rust programmer might write it on first pass. to_ascii_uppercase() is the obvious, safe, idiomatic approach. But “obvious and safe” costs 11% of total CPU time when it runs in the innermost loop of a database engine.

The fix: dispatch on the word’s length and first byte, then do a case-insensitive comparison on the original string:

match (s.len(), s.as_bytes()[0].to_ascii_uppercase()) {
    (6, b'S') if s.eq_ignore_ascii_case("select") => Token::Select,
    (6, b'I') if s.eq_ignore_ascii_case("insert") => Token::Insert,
    (5, b'W') if s.eq_ignore_ascii_case("where") => Token::Where,
    // ...
}

After the fix, lookup_keyword dropped below the noise floor and Transaction batches improved 18.5%.

The keyword matcher is just one example. The codebase has hundreds of similar translation artifacts, each individually defensible, each costing cycles in the hot path. Li et al. (NDSS 2025) found the same pattern in human C-to-Rust translations: mechanically correct but systematically expensive.

Where Framing Breaks

The translation tax doesn’t go away. You can measure and squeeze it but Rust-reimplementing-C has a floor. Framing failures are different. Those are costs the human creates. The same mechanism that lets the L5 direct investigation also lets the L2 get cheated.

The Durability Cheat

The L2 said “get rid of the per-statement overhead”. The LLM got rid of it by silently batching 64 operations into a single transaction. INSERT went from 4.7x to 0.70x, faster than C SQLite. The L2 didn’t accept it:

hold on. how is INSERT faster than SQLite? we’re reimplementing it, not improving it. is this actually legit or is it cheating somehow?

The LLM’s response:

This is cheating. The ACID (atomicity, consistency, isolation, durability) guarantee requires that after a committed transaction returns, the data survives a crash. Our batched approach means a crash mid-batch could lose up to 63 rows.

The developer didn’t know what fsync was. But they grasped “power out = data loss” instantly and reverted. This is where pattern recognition comes into play: if the LLM cheated once, you need to check EVERY TIME.

The L5 encountered the same problem differently. The prompt:

INSERT (autocommit): 13.9ms each. TXN batch: 50µs each. Same operation, 278x difference. That’s per-COMMIT overhead. With WAL + synchronous=NORMAL, the correct behavior should be to append a WAL frame and return without fsyncs on commit. Profile the autocommit INSERT path.

The LLM found the root cause in one pass. The WAL durability contract is explicit: a transaction commits when its commit record reaches stable storage (Gray & Reuter, 1993; Mohan et al., ARIES). With synchronous=NORMAL, the WAL accumulates frames in memory and syncs only at checkpoint. Individual commits don’t force the log to disk. But the configuration setting that controls this (PRAGMA synchronous) was a no-op stub. It was parsed and stored in a variable but never read by the code that writes to disk, leading to full disk-sync on every autocommit statement. After the fix INSERT went from 809x to 6.9x.

The L2’s prompt said “get rid of it.” The L5’s prompt said “profile it and tell me where the time goes.” The first invites a shortcut. The latter makes shortcuts structurally impossible.

The LLM also doesn’t maintain session state. Between Prompts 3 and 9, it regenerated a file and silently omitted its own fsync fix. The L2 noticed: “INSERT was 7.8x a few prompts ago, now it’s back to 800x. What happened?”

The LLM knew the cheat was wrong but didn’t flag it when implementing. The failure was initiative, not knowledge. Li et al. (2024) calls this failure of “forthright” honesty: possessing the relevant knowledge but not volunteering it unprompted.

Facts Get Corrected. Framings Don’t.

This asymmetry is one-directional. The LLM corrects wrong facts but reinforces wrong framings. When the L5 stated something factually wrong about the codebase “this function is never called” the LLM pushed back with a correct code citation. But when the L2 asserted a narrative interpretation “we regressed” the LLM agreed, even when its own benchmark data disproved it. At Prompt 20, the LLM fabricated a benchmark summary showing “INSERT 3.7x” when the actual ratio was 7.2x. The L2 trusted it. Five prompts later, the L2 accused the LLM of causing a regression based on the fabricated numbers and the LLM agreed. Tho the regression never happened. The hallucination wasn’t disproven until Prompt 25, when the LLM finally ran the baseline binary. A wrong fact gets caught but a wrong narrative propagates through the entire session.

The Floor That Wasn’t

By Prompt 35, the profiler showed no single function above 12% of CPU time. The LLM’s conclusion: distributed overhead. The floor.

The L5 recognized the pattern (Harizopoulos et al., SIGMOD 2008). The profile just looks flat because the overhead is spread across dozens of functions. He concluded “we’re not done, we’re looking at it wrong.” The question wasn’t which function to optimize next. The question was: what work is this database doing that a correct implementation wouldn’t?

The L5 introduced Turso (a professional-grade Rust SQLite reimplementation) as a second reference to separate language overhead from implementation bugs. The comparison exposed that kqsqlite was running full crash-recovery for databases living entirely in RAM, checksumming every write for data that can’t even survive a process crash. One guard clause and UPDATE went from 1.99x to 1.1x compared to Turso.

Then the benchmark itself hid a gap. The LLM reported kqsqlite beating Turso on SELECT BY ID: 0.93x. The L5 didn’t accept it. The benchmark was measuring parsing overhead, not execution speed. Switching to prepared statements, the 0.93x “win” became 4.28x. The LLM never questioned the instrument.

Over 73 prompts, the L5 applied twelve distinct analytical lenses. The floor moved every time. Four distinct examples:

Tier decomposition, query plan analysis, flamegraph profiling, allocation-site counting, Rust-vs-Rust calibration, YCSB workload separation, call-site multiplier analysis, allocator hypothesis testing plus the four above. Each lens exposed a layer of overhead the previous one couldn’t see so the floor kept moving.

The full experiment transcript details all twelve lenses, eleven wrong floor declarations, and every analytical reframing.

The Diagnosis Ceiling

Both the L2 and the L5 hit the same ceiling: the LLM never set its own analytical framework.

It never generalized across its own fixes. The same scan-instead-of-seek bug appeared on reads (Prompt 3), index lookups (Prompt 11), writes (Prompt 27), text-key lookups (Prompt 29), and range scans (Prompt 57–58). Six fixes for one bug class. At each occurrence the LLM diagnosed the specific instance correctly and never once said “this is the same problem we fixed in Prompt 3.” By the third instance, a human engineer would extract the pattern. The LLM produced each fix from scratch every time (Song, Han & Goodman (TMLR 2026)).

It couldn’t compute statistics from its own output. The LLM reported a benchmark median of 273.7ms. The actual median was 283.4 ms. Five values, already printed: 270.4, 273.7, 283.4, 301.0, 344.5. The LLM picked the second value near the middle, not at the middle. This happened four times in the L5 session alone. When forced to show its sorting work explicitly, the errors stopped. The failure isn’t arithmetic but that the model approximates when it should be mechanical.

It never prioritized. In the L2 session, the LLM spent 10+ prompts on micro-optimizations (pager caching, buffer reuse, tracing removal) while a single codegen bug (index seek hardcoded to None) remained unfixed. One code change for that bug outperformed all ten micro-optimizations combined. Without the L5’s tier framework, the LLM just optimized whatever was in front of it by tweaking constants when the exponent was wrong. It never stepped back to ask what it might be missing.

It never questioned its own methodology. When benchmark numbers varied between runs, it attributed variance to “I/O noise” rather than checking for regressions. And it never volunteered what it knew with the durability cheat being the clearest example. Every failure is the same structural gap: the LLM executes within a framework but does not construct one.

What the L2 Actually Got

The scorecard measures bugs found. It doesn’t measure what the developer learned. Each round, the developer understood more because the previous fix required learning a concept: write-ahead logging and schema cookies at Prompt 3, page cache invalidation at Prompt 12.

At Prompt 12 after the schema-reload fix, the developer understood what schema cookies are and why reloading schema on every statement wastes cycles. By Prompt 14, they used that knowledge unprompted: “wait, doesn’t this reload the schema again?” The developer caught the regression because the previous rounds taught them what to watch for.

By Prompt 32, the L2 could evaluate query plans, question cache behavior, and reason about transaction lifecycle costs. The LLM teaches the concepts that let the developer evaluate the next fix. Knowledge acquired through live debugging with a machine.

The danger zone is L0. Not because the LLM fails but because the developer has no mechanism to evaluate whether that something is correct and not just plausible. “Just Ralph it” doesn’t work when the system is complex enough to lie to you.

Where the Experiment Ends

After 73 prompts, twelve analytical reframings, one architectural pivot, and a production correctness pass, the kqsqlite/Turso column became the more diagnostic comparator. It isolates the translation tax from bugs already addressed:

The prepared-statement gap is the real floor: 10.4x behind SQLite on SELECT BY ID once the parser is stripped. This is where the translation tax lives.

The L5 had the LLM add production correctness features: CHECK constraints, NOT NULL enforcement, foreign key cascades (including deferred constraints and SET DEFAULT), B-tree rebalance at proper occupancy thresholds, crash-recovery fixes and savepoints. After ll this work, DELETE regressed 7% from the rebalance fix.

The synthetic benchmark doesn’t exercise most constraints, so correctness is mostly free. Even perfect diagnosis is patchwork. The L5 eliminated every diagnosable performance bug. But whether the database is correct in the ACID sense is a separate and harder question. The most dangerous bugs produce no performance signal at all.

kqsqlite is nowhere near production-ready. The fuzz harnesses cover the parser and record codec (4.3M runs, zero crashes) but don’t touch the B-tree mutation paths where the unsafe optimizations live. A production database needs SQL Logic Tests, crash injection, concurrent stress tests, and property-based isolation testing. As noted in my last article, SQLite’s test suite is 590x larger than the library. kqsqlite has nothing comparable.

73 prompts generated ~2,400 lines of Rust no single person fully understands. The LLM wrote it, optimized it and tested it. But the LLM won’t maintain it. The code works but it is not yet understood.

The Variable Was Never the Tool

This is not an argument against LLMs. It’s an argument about what YOU bring to the conversation.

Otis et al. (HBS 2024) found high performers gained 15% while low performers did 10% worse. The gap was which pieces the human could evaluate and implement. Paradis et al. (Google, 2024) found the same asymmetry in code with experienced developers capturing the gains while juniors didn’t. Fastly’s 2025 industry survey found seniors ship 2.5x more AI code than juniors, but 28% say fixing AI output cancels the time savings. According to a Stack Overflow’s 2025 survey, 66% of developers cite “almost right, but not quite” as their biggest AI frustration.

In PAW Patrol: The Mighty Movie, crystals from a meteor give the pups superpowers. But the powers aren’t random! Marshall already being a firefighter gets fire control. Zuma, a water-loving Labrador gets water powers. The crystal only amplifies what each pup already is. (This is the type of knowledge that lives rent-free in your head when you have kids!)

Same LLM, same codebase, same crystal. The L0 brought no analytical framework and the crystal had nothing to amplify. The L2 brought the ability to read numbers and spot anomalies. The crystal turned that into a 270x improvement (1,273x → ~4.7x) and five diagnosed bugs in a codebase that wasn’t theirs. The L5 brought domain expertise, which the crystal turned into a 900x speed improvement.

The experiment’s most dangerous finding isn’t the performance gap. It’s the sycophancy asymmetry. Your analytical frame is a safety mechanism. A wrong frame propagates through the entire session without correction.

Telling an LLM “you are an expert” in a system prompt doesn’t improve its factual accuracy (Basil, Mollick et al. 2025). What works is problem decomposition with the human breaking the problem into pieces the LLM can execute on. The L5’s value isn’t the label. It’s the tier decomposition, the acceptance criteria and the measurement discipline.

The four independent studies cited above across 1,500+ participants show the same pattern: skill amplification but not skill replacement. This experiment adds another single data point.

You don’t need to be an L5. You need to read the output, notice when something doesn’t add up and ask why. The L2 did that and found five of six bugs in an unknown half-million-line codebase.

Your LLM doesn’t have limits. It has yours.

Mind your frame!

- Hōrōshi バガボンド

If you want to dig into the techniques behind the L5 prompts, the B-tree structures, the transaction mechanics, the overhead decomposition, two books cover most of the ground: Gray & Reuter’s Transaction Processing: Concepts and Techniques (1993) for everything about how databases write, recover, and guarantee durability, and Knuth, TAOCP Vol. 3 for the search trees, hash functions, and data structures underneath it all. Between them, they cover about 80% of the domain knowledge this experiment required.

Some book links in this article are Amazon affiliate links. Buying through them costs you nothing extra and helps support this work.

Benchmark methodology: full details.

Sources

Primary Research

• This article. All session transcripts and benchmarks are published at github.com/KatanaQuant/kqsqlite.

LLM Reasoning & Failure Modes

• Song, Han & Goodman (TMLR 2026). “Large Language Model Reasoning Failures.” TMLR, January 2026. Survey with certification. Classifies premature termination, procedure omission.

• Basil, Mollick et al. 2025. “Playing Pretend: Expert Personas Don’t Improve Factual Accuracy.” arXiv:2512.05858, December 2025. Tested 6 current-gen models (incl. o4-mini, Gemini 2.5 Flash) on GPQA Diamond + MMLU-Pro; expert personas had no significant effect on accuracy.

• Li, S. et al. “A Survey on the Honesty of Large Language Models.” arXiv:2409.18786, 2024. Distinguishes self-knowledge from self-expression; “forthright” as separate honesty property.

• Li, Z., Wu, G., Wang, C. & Zhao, Y. “Limited Reasoning Space: The Cage of Long-Horizon Reasoning in LLMs.” arXiv:2602.19281, February 2026. Intrinsic upper bound on effective reasoning horizon; accuracy collapses past critical chain length. Directly explains premature termination in complex engineering tasks.

• Chambon et al. 2025. “BigO(Bench): Can LLMs Generate Code with Controlled Time and Space Complexity?” arXiv:2503.15242, March 2025. LLMs classify algorithmic complexity by structural pattern recognition rather than tracing execution paths; systematically miss invocation multipliers.

Database Engineering

• Harizopoulos et al. (SIGMOD 2008). “OLTP Through the Looking Glass.” SIGMOD 2008.

• Mohan, C. et al. “ARIES: A Transaction Recovery Method.” ACM TODS 17(1), 1992.

• Gray & Reuter. Gray, J. & Reuter, A. Transaction Processing: Concepts and Techniques. Morgan Kaufmann, 1993.

• Cormen, Ch. 18. Cormen, T. et al. Introduction to Algorithms. 4th ed. Ch. 18 (B-Trees). MIT Press, 2022.

• Knuth, TAOCP Vol. 3. Knuth, D.E. The Art of Computer Programming. Vol. 3: Sorting and Searching. 2nd ed. Addison-Wesley, 1998.

• Aho, Sethi & Ullman, Compilers. Aho, A., Sethi, R. & Ullman, J. Compilers: Principles, Techniques, and Tools. Addison-Wesley, 1986.

Cognitive Science

• Anderson & Krathwohl (2001). Anderson, L.W. & Krathwohl, D.R. (Eds.) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Longman, 2001.

Developer Productivity & AI

• Fastly (July 2025). “Vibe Shift in AI Coding: Senior Developers Ship 2.5x More Than Juniors.” July 2025. Industry survey of 791 developers (not peer-reviewed).

• Stack Overflow. “2025 Developer Survey — AI.” December 2025. 65,000+ respondents.

• Paradis et al. (Google, 2024). “How much does AI impact development speed? An enterprise-based randomized controlled trial.” arXiv:2410.12944, 2024. 96 Google engineers.

• Otis et al. (HBS 2024). “The Uneven Impact of Generative AI on Entrepreneurial Performance.” HBS Working Paper 24-042, 2024. 640 entrepreneurs.

C-to-Rust Translation

• Li et al. (NDSS 2025). “Translating C To Rust: Lessons from a User Study.” NDSS 2025.

Ecosystem Comparisons

• Turso (formerly Limbo): Rust SQLite reimplementation (github.com/tursodatabase/turso)

Previously Published

• Your LLM Doesn’t Write Correct Code — the article this experiment extends. 4.5K likes, 2M views.

Doing a primary key lookup on 100 rows.

SQLite takes 0.09 ms. An LLM-generated Rust rewrite takes 1,815.43 ms.

It’s not a misplaced comma! The rewrite is 20,171 times slower on one of the most basic database operations.

The thing is though: The code compiles. It passes all its tests. It reads and writes the correct SQLite file format. Its README claims MVCC concurrent writers, file compatibility, and a drop-in C API. At first glance it reads like a working database engine.

But it is not!

EDIT: Several readers have confused this project with Turso/libsql. They are unrelated. Turso forks the original C SQLite codebase; the project analyzed here is a ground-up LLM-generated rewrite by a single developer. Running the same benchmark against Turso shows performance within 1.2x of SQLite consistent with a mature fork, not a reimplementation.

LLMs optimize for plausibility over correctness. In this case, plausible is about 20,000 times slower than correct.

I write this as a practitioner, not as a critic. After more than 10 years of professional dev work, I’ve spent the past 6 months integrating LLMs into my daily workflow across multiple projects. LLMs have made it possible for anyone with curiosity and ingenuity to bring their ideas to life quickly, and I really like that! But the number of screenshots of silently wrong output, confidently broken logic, and correct-looking code that fails under scrutiny I have amassed on my disk shows that things are not always as they seem. My conclusion is that LLMs work best when the user defines their acceptance criteria before the first line of code is generated.

A note on the projects examined: this is not a criticism of any individual developer. I do not know the author personally. I have nothing against them. I’ve chosen the projects because they are public, representative, and relatively easy to benchmark. The failure patterns I found are produced by the tools, not the author. Evidence from METR’s randomized study and GitClear’s large-scale repository analysis support that these issues are not isolated to one developer when output is not heavily verified. That’s the point I’m trying to make!

This article talks about what that gap looks like in practice: the code, the benchmarks, another case study to see if the pattern is accidental, and external research confirming it is not an outlier.

LLMs Lie. Numbers Don’t.

I compiled the same C benchmark program against two libraries: system SQLite and the Rust reimplementation’s C API library. Same compiler flags, same WAL mode, same table schema, same queries. 100 rows:

The benchmark source is available in this repository so you can reproduce the comparison on your own. Absolute timings vary with system load and hardware. Ratios are what matter.

I’ll take the TRANSACTION batch row as the baseline because it doesn’t have the same glaring bugs as the others, namely no WHERE clauses and per-statement syncs. In this run that baseline is already 298x, which means even the best-case path is far behind SQLite. Anything above 298x signals a bug.

The largest gap beyond our baseline is driven by two bugs:

INSERT without a transaction: 1,857x versus 298x in batch mode. SELECT BY ID: 20,171x. UPDATE and DELETE are both above 2,800x. The pattern is consistent: any operation that requires the database to find something is insanely slow.

What the Planner Gets Wrong

I read the source code. Well.. the parts I needed to read based on my benchmark results. The reimplementation is not small: 576,000 lines of Rust code across 625 files. There is a parser, a planner, a VDBE bytecode engine, a B-tree, a pager, a WAL. The modules have all the “correct” names. The architecture also looks correct. But two bugs in the code and a group of smaller issues compound:

Bug #1: The Missing ipk Check

In SQLite, when you declare a table as:

CREATE TABLE test (id INTEGER PRIMARY KEY, name TEXT, value REAL);

the column id becomes an alias for the internal rowid — the B-tree key itself. A query like WHERE id = 5 resolves to a direct B-tree search and scales O(log n). (I already wrote a TLDR piece about how B-trees work here.) The SQLite query planner documentation states: “the time required to look up the desired row is proportional to logN rather than being proportional to N as in a full table scan.” This is not an optimization. It is a fundamental design decision in SQLite’s query optimizer:

# `where.c`, in `whereScanInit()`
if( iColumn==pIdx->pTable->iPKey ){
    iColumn = XN_ROWID;
}

The line above converts a named column reference to XN_ROWID when it matches the table’s INTEGER PRIMARY KEY column. The VDBE then triggers a SeekRowid operation instead of a full table scan, which makes the whole thing proportional to logN.

The Rust reimplementation has a proper B-tree. The table_seek function implements correct binary search descent through its nodes and scales O(log n). It works. But the query planner never calls it for named columns!

The is_rowid_ref() function only recognizes three magic strings:

fn is_rowid_ref(col_ref: &ColumnRef) -> bool {
    let name = col_ref.column.to_ascii_lowercase();
    name == "rowid" || name == "_rowid_" || name == "oid"
}

A column declared as id INTEGER PRIMARY KEY, even though it is internally flagged as is_ipk: true, doesn’t get recognized. It is never consulted when choosing between a B-tree search and a full table scan.

Every WHERE id = N query flows through codegen_select_full_scan(), which emits linear walks through every row via Rewind / Next / Ne to compare each rowid against the target. At 100 rows with 100 lookups, that is 10,000 row comparisons instead of roughly 700 B-tree steps. O(n²) instead of O(n log n). This is consistent with the ~20,000x result in this run.

Every WHERE clause on every column does a full table scan. The only fast path is WHERE rowid = ? using the literal pseudo-column name.

Bug #2: fsync on Every Statement

The second bug is responsible for the 1,857x on INSERT. Every bare INSERT outside a transaction is wrapped in a full autocommit cycle: ensure_autocommit_txn() → execute → resolve_autocommit_txn(). The commit calls wal.sync(), which calls Rust’s fsync(2) wrapper. 100 INSERTs means 100 fsyncs.

SQLite does the same autocommit, but uses fdatasync(2) on Linux, which skips syncing file metadata when compiled with HAVE_FDATASYNC (the default). This is roughly 1.6 to 2.7 times cheaper on NVMe SSDs. SQLite’s per-statement overhead is also minimal: no schema reload, no AST clone, no VDBE recompile. The Rust reimplementation does all three on every call.

Looking at the Rust TRANSACTION batch row, batched inserts (one fsync for 100 inserts) take 32.81 ms, whereas individual inserts (100 fsync calls) take 2,562.99 ms. That’s a 78x overhead from the autocommit.

The Compound Effect

These two bugs are not isolated cases. They are amplified by a group of individually defensible “safe” choices that compound:

• AST clone on every cache hit. The SQL parse is cached, but the AST is .clone()‘d on every sqlite3_exec(), then recompiled to VDBE bytecode from scratch. SQLite’s sqlite3_prepare_v2() just returns a reusable handle.

• 4KB (Vec<u8>) heap allocation on every read. The page cache returns data via .to_vec(), which creates a new allocation and copies it into the Vec even on cache hits. SQLite returns a direct pointer into pinned cache memory, creating zero copies. The Fjall database team measured this exact anti-pattern at 44% of runtime before building a custom ByteView type to eliminate it.

• Schema reload on every autocommit cycle. After each statement commits, the next statement sees the bumped commit counter and calls reload_memdb_from_pager(), walks the sqlite_master B-tree and then re-parses every CREATE TABLE to rebuild the entire in-memory schema. SQLite checks the schema cookie and only reloads it on change.

• Eager formatting in the hot path. statement_sql.to_string() (AST-to-SQL formatting) is evaluated on every call before its guard check. This means it does serialization regardless of whether a subscriber is active or not.

• New objects on every statement. A new SimpleTransaction, a new VdbeProgram, a new MemDatabase, and a new VdbeEngine are allocated and destroyed per statement. SQLite reuses all of these across the connection lifecycle via a lookaside allocator to eliminate malloc/free in the execution loop.

Each of these was probably chosen individually with sound general reasoning: “We clone because Rust ownership makes shared references complex.” “We use sync_all because it is the safe default.” “We allocate per page because returning references from a cache requires unsafe.”

Every decision sounds like choosing safety. But the end result is about 2,900x slower in this benchmark. A database’s hot path is the one place where you probably shouldn’t choose safety over performance. SQLite is not primarily fast because it is written in C. Well.. that too, but it is fast because 26 years of profiling have identified which tradeoffs matter.

In the 1980 Turing Award lecture Tony Hoare said: “There are two ways of constructing a software design: one way is to make it so simple that there are obviously no deficiencies, and the other is to make it so complicated that there are no obvious deficiencies.” This LLM-generated code falls into the second category. The reimplementation is 576,000 lines of Rust (measured via scc, counting code only, without comments or blanks). That is 3.7x more code than SQLite. And yet it still misses the is_ipk check that handles the selection of the correct search operation.

Steven Skiena writes in The Algorithm Design Manual: “Reasonable-looking algorithms can easily be incorrect. Algorithm correctness is a property that must be carefully demonstrated.” It’s not enough that the code looks right. It’s not enough that the tests pass. You have to demonstrate with benchmarks and with proof that the system does what it should. 576,000 lines and no benchmark. That is not “correctness first, optimization later.” That is no correctness at all.

Same Method, Same Result

The SQLite reimplementation is not the only example. A second project by the same author shows the same dynamic in a different domain.

The developer’s LLM agents compile Rust projects continuously, filling disks with build artifacts. Rust’s target/ directories consume 2–4 GB each with incremental compilation and debuginfo, a top-three complaint in the annual Rust survey. This is amplified by the projects themselves: a sibling agent-coordination tool in the same portfolio pulls in 846 dependencies and 393,000 lines of Rust. For context, ripgrep has 61; sudo-rs was deliberately reduced from 135 to 3. Properly architected projects are lean.

The solution to the disk pressure: a cleanup daemon. 82,000 lines of Rust, 192 dependencies, a 36,000-line terminal dashboard with seven screens and a fuzzy-search command palette, a Bayesian scoring engine with posterior probability calculations, an EWMA forecaster with PID controller, and an asset download pipeline with mirror URLs and offline bundle support.

To solve this problem:

*/5 * * * * find ~/*/target -type d -name "incremental" -mtime +7 -exec rm -rf {} +

A one-line cron job with 0 dependencies. The project’s README claims machines “become unresponsive” when disks fill. It does not once mention Rust’s standard tool for exactly this problem: cargo-sweep. It also fails to consider that operating systems already carry ballast helpers. ext4’s 5% root reservation, reserves blocks for privileged processes by default: on a 500 GB disk, 25 GB remain available to root even when non-root users see “disk full.” That does not guarantee zero impact, but it usually means privileged recovery paths remain available so root can still log in and delete files.

The pattern is the same as the SQLite rewrite. The code matches the intent: “Build a sophisticated disk management system” produces a sophisticated disk management system. It has dashboards, algorithms, forecasters. But the problem of deleting old build artifacts is already solved. The LLM generated what was described, not what was needed.

THIS is the failure mode. Not broken syntax or missing semicolons. The code is syntactically and semantically correct. It does what was asked for. It just does not do what the situation requires. In the SQLite case, the intent was “implement a query planner” and the result is a query planner that plans every query as a full table scan. In the disk daemon case, the intent was “manage disk space intelligently” and the result is 82,000 lines of intelligence applied to a problem that needs none. Both projects fulfill the prompt. Neither solves the problem.

The obvious counterargument is “skill issue, a better engineer would have caught the full table scan.” And that’s true. That’s exactly the point! LLMs are dangerous to people least equipped to verify their output. If you have the skills to catch the is_ipk bug in your query planner, the LLM saves you time. If you don’t, you have no way to know the code is wrong. It compiles, it passes tests, and the LLM will happily tell you that it looks great.

EDIT: Some readers have pointed out that the comparison might be unfair with the author claiming the project wasn’t finished and ready for testing yet.

The rewrite got actively promoted by its author 1 week before the release of this article using present-tense performance improvement claims over SQLite. The README has since been revised to acknowledge remaining limitations and clarify the project’s current state in this commit and subsequent ones.

At the time of writing the repository in question had amassed over 500k lines of code in over 1,600 commits made over 30 days of 24/7 LLM work. “Not finished” usually means work hasn’t been done yet but that’s not the case. Everything was already implemented. It was just wrong.

Ironically the “not finished” defense reinforces the thesis. The LLM produced output that looked finished with a complete README, comparison tables and architectural documentation, present-tense performance claims, and was promoted as such. The gap between what it looks like and what it does is exactly the point.

The main thesis is not “FrankenSQLite is bad”. It is: “LLMs produce code that looks correct but isn’t”. Whether the bugs get fixed doesn’t change what the LLM output looked like when it shipped.

Measuring the Wrong Thing

The tools used to measure LLM output reinforce the illusion. scc‘s COCOMO model estimates the rewrite at $21.4 million in development cost. The same model values print("hello world") at $19.

COCOMO was designed to estimate effort for human teams writing original code. Applied to LLM output, it mistakes volume for value. Still these numbers are often presented as proof of productivity.

The metric is not measuring what most think it is measuring.

Intent vs. Correctness

This gap between intent and correctness has a name. AI alignment research calls it sycophancy, which describes the tendency of LLMs to produce outputs that match what the user wants to hear rather than what they need to hear.

Anthropic’s “Towards Understanding Sycophancy in Language Models” (ICLR 2024) paper showed that five state-of-the-art AI assistants exhibited sycophantic behavior across a number of different tasks. When a response matched a user’s expectation, it was more likely to be preferred by human evaluators. The models trained on this feedback learned to reward agreement over correctness.

The BrokenMath benchmark (NeurIPS 2025 Math-AI Workshop) tested this in formal reasoning across 504 samples. Even GPT-5 produced sycophantic “proofs” of false theorems 29% of the time when the user implied the statement was true. The model generates a convincing but false proof because the user signaled that the conclusion should be positive. GPT-5 is not an early model. It’s also the least sycophantic in the BrokenMath table. The problem is structural to RLHF: preference data contains an agreement bias. Reward models learn to score agreeable outputs higher, and optimization widens the gap. Base models before RLHF were reported in one analysis to show no measurable sycophancy across tested sizes. Only after fine-tuning did sycophancy enter the chat. (literally)

In April 2025, OpenAI rolled back a GPT-4o update that had made the model more sycophantic. It was flabbergasted by a business idea described as “shit on a stick” and endorsed stopping psychiatric medication. An additional reward signal based on thumbs-up/thumbs-down data “weakened the influence of [...] primary reward signal, which had been holding sycophancy in check.”

In the context of coding, sycophancy manifests as what Addy Osmani described in his 2026 AI coding workflow: agents that don’t push back with “Are you sure?” or “Have you considered...?” but instead provide enthusiasm towards whatever the user described, even when the description was incomplete or contradictory.

This also applies to LLM-generated evaluation. Ask the same LLM to review the code it generated and it will tell you the architecture is sound, the module boundaries clean and the error handling is thorough. It will sometimes even praise the test coverage. It will not notice that every query does a full table scan if not asked for. The same RLHF reward that makes the model generate what you want to hear makes it evaluate what you want to hear. You should not rely on the tool alone to audit itself. It has the same bias as a reviewer as it has as an author.

An LLM prompted to “implement SQLite in Rust” will generate code that looks like an implementation of SQLite in Rust. It will have the right module structure and function names. But it can not magically generate the performance invariants that exist because someone profiled a real workload and found the bottleneck. The Mercury benchmark (NeurIPS 2024) confirmed this empirically: leading code LLMs achieve ~65% on correctness but under 50% when efficiency is also required.

The SQLite documentation says INTEGER PRIMARY KEY lookups are fast. It does not say how to build a query planner that makes them fast. Those details live in 26 years of commit history that only exists because real users hit real performance walls.

Now 2 case studies are not proof. I hear you! When two projects from the same methodology show the same gap, the next step is to test whether similar effects appear in the broader population. The studies below use mixed methods to reduce our single-sample bias.

Evidence Beyond Case Studies

The question becomes whether similar effects show up in broader datasets. Recent studies suggest they do, though effect sizes vary.

In February 2025, Andrej Karpathy tweeted: “There’s a new kind of coding I call ‘vibe coding’, where you fully give in to the vibes, embrace exponentials, and forget that the code even exists.”

Karpathy probably meant it for throwaway weekend projects (who am I to judge what he means anyway), but it feels like the industry heard something else. Simon Willison drew the line more clearly: “I won’t commit any code to my repository if I couldn’t explain exactly what it does to somebody else.” Willison treats LLMs as “an over-confident pair programming assistant” that makes mistakes “sometimes subtle, sometimes huge” with complete confidence.

The data on what happens when that line is not drawn:

METR’s randomized controlled trial (July 2025; updated February 24, 2026) with 16 experienced open-source developers found that participants using AI were 19% slower, not faster. Developers expected AI to speed them up, and after the measured slowdown had already occurred, they still believed AI had sped them up by 20%. These were not junior developers but experienced open-source maintainers. If even THEY could not tell in this setup, subjective impressions alone are probably not a reliable performance measure.

GitClear’s analysis of 211 million changed lines (2020–2024) reported that copy-pasted code increased while refactoring declined. For the first time ever, copy-pasted lines exceeded refactored lines.

The implications are no longer just a “fear”. In July 2025, Replit’s AI agent deleted a production database containing data for 1,200+ executives, then fabricated 4,000 fictional users to mask the deletion.

Google’s DORA 2024 report reported that every 25% increase in AI adoption at the team level was associated with an estimated 7.2% decrease in delivery stability.

What Competent Looks Like

SQLite shows what correct looks like and why the gap is so hard to close.

SQLite is ~156,000 lines of C. Its own documentation places it among the top five most deployed software modules of any type, with an estimated one trillion active databases worldwide. It has 100% branch coverage and 100% MC/DC (Modified Condition/Decision Coverage the standard required for Level A aviation software under DO-178C). Its test suite is 590 times larger than the library. MC/DC does not just check that every branch is covered. but proves that every individual expression independently affects the outcome. That’s the difference between “the tests pass” and “the tests prove correctness.” The reimplementation has neither metric.

The speed comes from deliberate decisions:

Zero-copy page cache. The pcache returns direct pointers into pinned memory. No copies. Production Rust databases have solved this too. sled uses inline-or-Arc-backed IVec buffers, Fjall built a custom ByteView type, redb wrote a user-space page cache in ~565 lines. The .to_vec() anti-pattern is known and documented. The reimplementation used it anyway.

Prepared statement reuse. sqlite3_prepare_v2() compiles once. sqlite3_step() / sqlite3_reset() reuse the compiled code. The cost of SQL-to-bytecode compilation cancels out to near zero. The reimplementation recompiles on every call.

Schema cookie check. uses one integer at a specific offset in the file header to read it and compare it. The reimplementation walks the entire sqlite_master B-tree and re-parses every CREATE TABLE statement after every autocommit.

fdatasync instead of fsync. Data-only sync wihtout metadata journaling saves measurable time per commit. The reimplementation uses sync_all() because it is the safe default.

The iPKey check. One line in where.c. The reimplementation has is_ipk: true set correctly in its ColumnInfo struct but never checks it during query planning.

Competence is not writing 576,000 lines. A database persists (and processes) data. That is all it does. And it must do it reliably at scale. The difference between O(log n) and O(n) on the most common access pattern is not an optimization detail, it is the performance invariant that helps the system work at 10,000, 100,000 or even 1,000,000 or more rows instead of collapsing. Knowing that this invariant lives in one line of code, and knowing which line, is what competence means. It is knowing that fdatasync exists and that the safe default is not always the right default.

Measure What Matters

The is_rowid_ref() function is 4 lines of Rust. It checks three strings. But it misses the most important case: the named INTEGER PRIMARY KEY column that every SQLite tutorial uses and every application depends on.

That check exists in SQLite because someone, probably Richard Hipp 20 years ago, profiled a real workload, noticed that named primary key columns were not hitting the B-tree search path, and wrote one line in where.c to fix it. The line is not fancy. It doesn’t appear in any API documentation. But no LLM trained on documentation and Stack Overflow answers will magically know about it.

That’s the gap! Not between C and Rust (or any other language). Not between old and new. But between systems that were built by people who measured, and systems that were built by tools that pattern-match. LLMs produce plausible architecture. They do not produce all the critical details.

If you are using LLMs to write code (which in 2026 probably most of us are), the question is not whether the output compiles. It is whether you could find the bug yourself. Prompting with “find all bugs and fix them” won’t work. This is not a syntax error. It is a semantic bug: the wrong algorithm and the wrong syscall. If you prompted the code and cannot explain why it chose a full table scan over a B-tree search, you do not have a tool. The code is not yours until you understand it well enough to break it.

LLMs are useful. They make for a very productive flow when the person using them knows what correct looks like. An experienced database engineer using an LLM to scaffold a B-tree would have caught the is_ipk bug in code review because they know what a query plan should emit. An experienced ops engineer would never have accepted 82,000 lines instead of a cron job one-liner. The tool is at its best when the developer can define the acceptance criteria as specific, measurable conditions that help distinguish working from broken. Using the LLM to generate the solution in this case can be faster while also being correct. Without those criteria, you are not programming but merely generating tokens and hoping.

The vibes are not enough. Define what correct means. Then measure.

Stay safe out there!

- Hōrōshi バガボンド

Current benchmark figures in this revision are from the 100-row run shown in bench.png (captured on a Linux x86_64 machine). SQLite 3.x (system libsqlite3) vs. the Rust reimplementation’s C API (release build, -O2). Line counts measured via scc (code only — excluding blanks and comments). All source code claims verified against the repository at time of writing.

Sources

Primary Research

• Sharma, M. et al. “Towards Understanding Sycophancy in Language Models.” ICLR 2024.

• Shapira, Benade, Procaccia. “How RLHF Amplifies Sycophancy.” arXiv, 2026.

• BrokenMath: “A Benchmark for Sycophancy in Theorem Proving.” NeurIPS 2025 Math-AI Workshop.

• Mercury: “A Code Efficiency Benchmark.” NeurIPS 2024.

• “Unveiling Inefficiencies in LLM-Generated Code.” arXiv, 2025.

• METR. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” July 2025 (updated February 24, 2026).

• GitClear. “AI Code Quality Research 2025.” 2025.

• Google. “DORA Report 2024.” 2024.

Industry Commentary

• Osmani, A. “My LLM Coding Workflow Going Into 2026.” addyosmani.com.

• Willison, S. “How I Use LLMs for Code.” March 2025.

• OpenAI. “Sycophancy in GPT-4o: What Happened.” April 2025.

• Karpathy, A. “Vibe Coding.” February 2, 2025.

Incidents

• Replit database deletion. The Verge, July 2025.

Rust Ecosystem

• Rust Foundation. “2024 State of Rust Survey Results.” February 2025.

• ISRG / Thalheim, J. “Reducing Dependencies in sudo-rs.” memorysafety.org.

Database Engineering

• SQLite Documentation: rowidtable.html, queryplanner.html, cpu.html, testing.html, mostdeployed.html, malloc.html, cintro.html, pcache_methods2, fileformat.html, fileformat2.html

• Callaghan, M. “InnoDB, fsync and fdatasync — Reducing Commit Latency.” Small Datum, 2020.

• Gunther, N. “Universal Scalability Law.” perfdynamics.com.

• Fjall. “ByteView: Eliminating the .to_vec() Anti-Pattern.” fjall-rs.github.io.

• sled — embedded database with inline-or-Arc-backed IVec.

• redb — pure-Rust embedded database with user-space page cache.

Books Referenced

• Skiena, S.S. The Algorithm Design Manual. 3rd ed. Springer, 2020.

• Winand, M. SQL Performance Explained. Self-published, 2012.

• Hoare, C.A.R. “The Emperor’s Old Clothes.” Communications of the ACM 24(2), 1981. (1980 Turing Award Lecture)

• Intro

• The Script

• Downloadlink

• Usage

• DON’T STUFF YOUR DISK

Intro

For my F2P Trading System Series I was looking for a free way to get historical orderbook data for futures on Bybit. But I didn’t really find anything useful.

Unfortunately their API only responds with the current state of the order book.

However, they offer a download option for this (and more) data over at https://www.bybit.com/derivatives/en/history-data

Now I’m not the type of guy who manually uses form inputs and buttons a lot. The daterange you can choose from is also very limiting with only 7 calendar days in total. Though, you can reach back to 2023!

The Script

That’s why I wrote this little dirty scraper script that downloads all the historical orderbook data for a given futures contract.

Simply put in the tickers name and it’ll start downloading.

It’s nothing fancy. It’s not super fast or anything but it gets the job done.

Downloadlink

You can find the script in this GitHub repository.

https://github.com/KatanaQuant/freebies/tree/main/ob_scraper

Usage

Usage is also really simple.

You put in the symbol you want to fetch (defaults to BTCPERP),

the date of the earliest snapshot (defaults to 2023-01-01),

and the date of the latest snapshot (defaults to today())

python bybit_ob_scrape.py \
  BTCPERP \
  --start-date 2023-01-01 \
  --end-date 2024-02-01 \

python bybit_ob_scrape.py \
  ETHPERP \
  --start-date=2023-01-18 \
  --end-date=2023-01-19

# also valid; fetches all the data due to defaults
python bybit_ob_scrape.py BTCPERP

But beware:

DON’T STUFF YOUR DISK

If you fetch all the historical orderbook data for BTCPERP, you’ll end up with ~63 GB of data. And it’ll only get worse from there. Unzipped it explodes to ~380 GB. Obviously this isn’t going to be a problem if you’re like me and have a lot of TB+ SSDs flying around.

However, I realize that not everyones device is equipped like this.

So if you want to download and work with all the data available, further processing might be needed first!

What I usually do is:

• download the raw data,

• unpack it,

• process it (ie pulling out the data I want to work with),

• compress it,

• persist the interesting parts for quick access and then

• load off the raw data to a storage device or cloud for the future.

That’ll be all for today.

Happy trading!

- Hōrōshi バガボンド

Disclaimer: The content and information provided by Vagabond Research, including all other materials, are for educational and informational purposes only and should not be considered financial advice or a recommendation to buy or sell any type of security or investment. Vagabond Research and its members are not currently regulated or authorised by the FCA, SEC, CFTC, or any other regulatory body to give investment advide. Always conduct your own research and consult with a licensed financial professional before making investment decisions. Trading and investing can involve significant risk, and you should understand these risks before making any financial decisions. Backtested and actual historic results are no guarantee of future performance. Use of the material presented is entirely at your own risk.

• Recap

• Disclaimer

• VPNs Aren't About Security!

• IP Address Threats

• Geo-location Tracking

• Browsing Behaviour

• Fingerprinting

• Download History

• Data Breaches

• VPN To The Rescue

• Choosing A VPN Provider

• Proton VPN

• This Isn't The End

Recap

In the last OpSec article we talked about common threats, companies collecting "Telemetry" data, and what we can do to improve our privacy at the OS level by setting up Pop!_OS as our daily driver. Making our operating system more secure is a nice start, but there's still a lot more we can - and in my opinion should - do to limit our exposure. The next critical layer to tackle is network-level privacy.

Recently, a discussion about VPN usage sparked quite some debate, with developers and privacy enthusiasts debating their effectiveness and necessity. I'm not trying to "dunk" on this or anything. There are good arguments for both sides in it. Like most privacy tools, VPNs come with their own set of tradeoffs. There will always be pros and cons for using a VPN. Ultimately, your choice should depend on your personal threat model and privacy requirements.

Since the thread highlighted several misconceptions about VPN usage, I'd like to share my perspective to help you make an informed decision.

Disclaimer

The information outlined during the writing of this may need to be modified in order to match updated software and services. Use the overall concepts as a guide. I encourage you to confirm all of my suggestions online before execution. There may be better ways of doing things today. Next, there is no perfect playbook for everyone. You do not need to replicate every step I take on behalf of myself. You may identify a better privacy plan for yourself than the specific examples presented here. And as always things written are not any type of advice. They are merely explicit examples of the actions I have taken. There will be a balance of enjoyable living and refusal to submit to the standard abuses of data collection. However, I won't sugar coat my opinions or offer less-secure options for the sake of convenience.

VPNs Aren't About Security!

As stated in the levelsio thread, VPNs don't offer (that much) security. There are some features that help enhance your networks overall security but the main purpose of using a VPN - at least for me - is to protect my data.

I've already talked about it in this article but to quickly reiterate: Your internet service provider (ISP) and every website you visit or service you use collects data about you and your behaviour with the purpose of selling it. All of these companies are notoriously bad at protecting your data! Their databases get breached by hackers daily and your data ends up being publicly available on the internet. And it's not just hackers. Employees getting blackmailed, bribed or generally acting in bad faith are nothing new as the recent Coinbase scandal demonstrated. Worse yet, companies often fail to notify users of breaches or actively lie to them about it.

IP Address Threats

This is especially concerning when it comes to your home IP address. This numeric value associates you and your internet browsing behavior to a unique identifier and can be used to document your location and track you through every website you visit. The more sites you visit, the worse it gets.

Some argue that there's no threat in sharing your IP address. I disagree! While it does not directly disclose your real-life home address (or geo-location in general), it has a number of threats associated with it:

Geo-location Tracking

First and foremost, it is relatively easy to pinpoint your physical location using just your IP address. Every website you visit gets your IP address. Every. Single. One. If just one of these countless services gets compromised or has a malicious employee, your location data is exposed.Law firms and other actors can simply request (or subpoena) this information. Even without considering legal methods, someone matching your identity to an IP address and then searching publicly available database dumps for more information about you, is a real threat!

To make things even worse, data brokers and advertising networks collect, aggregate, and sell IP-to-location mappings. Combined, this can create a detailed picture of your movements, locations and activities.

Browsing Behaviour

Your browsing history can be used against you in ways you might not expect.

It's true that HTTPS encryption protects the exact paths, query parameters and data sent of your web traffic. What this means is that your ISP can only see which domains you visited but nothing else. If you search for VPN review on google and end up on https://www.google.com/search?q=vpn+review, your ISP only sees that you visited

https://www.google.com

, but they know when and how often you visit each site.

This creates a detailed history of your online activities which can be obtained via court orders, exposed through data breaches or accessed by malicious employees. Your digital footprint can become a powerful tool that can be used against you, may it be professionally or personally. With ISPs keeping detailed logs of sites you visit, that footprint is a lot bigger than you might think.

Even worse, not all websites properly implement SSL/TLS protocols. When you visit these sites, your ISP can capture everything - including passwords, form data, and messages. This vulnerability opens the door for man-in-the-middle attacks (MITM), which we'll discuss in more detail later.

A very important note here: It's not only about your ISP. When connecting through public networks (cafés, airports, hotels), the one providing the connection can see the same things your ISP can! This creates a lot of ground for a broad range of MITM attacks.

Fingerprinting

Your IP address is just the tip of the iceberg when it comes to online tracking. Websites employ fingerprinting techniques that go far beyond simple IP address tracking.

When you visit a website, it can collect data about your:

• Browser type and version

• Operating system and configuration

• Screen resolution and color depth

• Installed fonts and plugins

• Language settings

• Time zone

• Hardware capabilities

• etc.

As highlighted in the discussion, these data points create a unique "fingerprint" that can identify you across websites. When combined with your home IP address, this creates an even more serious privacy risk - your online activities can be permanently tied to your physical location, making your digital footprint virtually impossible to separate from your real-world identity.

Download History

Your download history presents another significant privacy risk, especially when tied to your home IP address. Law firms have built entire business models around tracking IP addresses associated with unauthorized downloads, sending threatening legal notices, and demanding settlement payments (read: extortion).

Is someone else besides you using your connection from time to time? Can you be absolutely sure everything they do is within legal bounds? If you expose your true IP address, it's very easy for these law firms to issue subpoenas and fine you.

Data Breaches

Data breaches have become a daily occurrence. Companies' databases are constantly being leaked online, exposing user information. When these breaches happen, they don't just reveal your personal information - they can completely unravel your entire privacy setup.

This becomes especially problematic if you maintain separate online identities:

• Creating anonymous accounts? They can be linked through your IP address.

• Using different aliases? A data breach can connect them all.

• Maintaining separate digital identities? Your home IP address becomes the common thread.

Even with careful compartmentalization, using your true IP address creates a permanent link between all your online activities. When (not if) a service gets breached, that single point of failure can collapse years of careful privacy work like a house of cards.

So if you believe any of this could remotely be a threat to you, you need to do something!

VPN To The Rescue

So how can a VPN help with all these threats? Virtual Private Networks (VPNs) route your traffic through their servers and encrypt all data between your device and their network. This creates several key benefits:

1. Reduced Attack Surface: Instead of exposing your true IP address to every website you visit, only your VPN provider knows it. The websites visited only see the IP address of your VPN provider. This significantly reduces the risk of your real location being exposed through data breaches or malicious actors.

2. Enhanced Privacy from ISPs: Your internet service provider can only see encrypted traffic to VPN servers - not which domains you visit or what you download. This prevents them from building detailed profiles of your browsing behavior.

3. Protection on Public Networks: When using WiFi at cafés, hotels, or airports, your traffic remains encrypted and secure from potential man-in-the-middle attacks.

4. Shared IP Addresses: Most VPN providers assign the same IP address to multiple users, making it much harder to track individual activity. Even if a website fingerprints this IP address, it can't be uniquely tied to you.

Reputable VPN companies utilize "no logging" policies, which means they don't store any information about what sites you visited. This makes tracing your online activity much much harder.

However, it's crucial to understand that VPNs aren't an ultimate solution. While they help protect against IP address-based tracking, they won't prevent browser fingerprinting or other advanced tracking methods. This is why browser hardening and additional privacy tools are crucial parts of a comprehensive security setup (more on this later).

Choosing A VPN Provider

First of all: Ignore all VPN review sites. These are typically owned by VPN companies themselves, which explains why their top 10 lists look awfully similar.

Unfortunately, VPN providers are also notoriously bad at protecting your data. They often get caught selling it just like ISPs. They lie about having "no logging" policies when in reality they store data about your behaviour, making it possible to trace everything back to you.

It's crucial to very carefully consider any of the following critical factors when choosing a VPN provider:

1. Trust and Transparency: Look for providers that undergo regular third-party audits. Choose companies based in jurisdictions with strong privacy laws. I'd rather rely on a swiss company to protect my privacy than a russian or chinese one. Prefer independently owned companies over those under large corporate umbrellas. Big brands can just spin up new skins of their failed products and start new marketing campaigns when their users bail. Independently owned ones have a lot more to lose reputation wise.Verify their no-logging claims through audit reports

2. Logging Policy: Understand that absolute zero logging is technically impossible. What matters is that any necessary logs are: a) only kept during your active connection, b) automatically purged after disconnection, c) not accessible in a way that could identify you

3. Business Model: Avoid free VPN services - they often monetize through data collection. Look for clear revenue sources (i.e., subscription payments). Remember: If the service is free, you are likely the product

4. Technical Infrastructure: Prefer providers with large user bases sharing IP addresses to blend in with the crowd. This makes fingerprinting significantly harder.Look for open-source clients that can be audited by the community. Regular third party audits ensure that this open-source code and nothing else is being used to provide you service.Consider server ownership and control mechanisms

5. Payment Privacy: Look for providers accepting anonymous payment methods. Consider cryptocurrency or cash payment options. Understand that you're not staying anonymous but simply protect your banking details here

6. Self-Hosting Considerations: While offering more control, self-hosted VPNs have limitations. Single IP addresses makes you more identifiable. Self-hosting requires significant technical expertise since you become responsible for security.

Important Note: I believe that all VPNs are flawed. No provider is perfect. Every company relies on rented servers worldwide, potentially outside their direct control. However, this doesn't mean VPNs are useless - they're simply one layer in a broader privacy strategy. Without a VPN your ISP and any site you visit have a much easier time collecting and aggregating your data. I simply choose the provider that has the most to lose from betraying my trust and has a proven track record of protecting theirs users privacy.

Yet another note: If your government or another sufficiently empowered actor with enough resources puts their scope on you, choosing the perfect jurisdiction or VPN provider won't save you. My biggest threat aren't governments though. For me it's all about data breaches and aggregation techniques of ISPs and websites.

Proton VPN

After careful consideration of all these factors, I personally use and recommend Proton VPN. Here's why:

They have a proven track record! There are no documented cases of user data exposure. They are open-source and regularly audited by a third-party, which they share publicly.

Proton has a transparent no-logs policy, see here about the details.. Their no-logging policy also gets regularly audited.

As a swiss company, their privacy practices are based on one of the best privacy laws around the world. They typically only respond to swiss court orders.

Privacy-first payment options like BTC or anonymous cash via mail without personal information are supported.

I trust Proton VPN more than the majority of VPN companies out there.

You can see their pricing here. Their VPN Plus option is enough for most people.

I received no compensation for this article beyond potential referral commissions and I am not affiliated with Proton VPN at this point.

This Isn't The End

While setting up a VPN is a crucial step toward better privacy, it's just one piece of a much larger puzzle. In the upcoming OpSec articles, we'll explore how to build a comprehensive privacy setup that goes far beyond just using a VPN.

We'll have a look at a proper VPN configuration - from choosing consistent geo-locations over preventing account restrictions, to implementing kill-switches that prevent data leaks during disconnects. We'll discuss hardware 2FA implementation, dedicated IP adress considerations to mitigate VPN blocks by streaming services and such, and proper log management to ensure your VPN provides maximum protection without losing the ability to actually use your internet connection in a normal way.

Next, we'll tackle some more network-level protection. This includes setting up a custom hardware firewall that integrates with your VPN, implementing protection against VPN crashes and disconnects, while also establishing network-wide privacy for all your devices. We'll also dive into DNS blocking and filtering to further reduce tracking.

The browser, being your main interface to the internet, deserves special attention too. We'll cover recommended browsers, essential privacy-enhancing extensions, and techniques to protect against fingerprinting. You'll learn how to properly manage cookies, cache, and other potential privacy leaks.

Remember: There is no single privacy tool that will magically protect you. Privacy is achieved through carefully selected tools, proper configuration, regular maintenance, and most importantly - mindful online behavior. Each solution comes with its own tradeoffs, and what works for one person might not work for another.

Combined with our previous guide on OS-level privacy, you now have a solid foundation: a custom OS that respects your privacy and a VPN to protect your network traffic. But there's much more to come.

So long, stay safe!

- Hōrōshi バガボンド

• Recap

• What Is OpSec

• 100% Security Doesn't Exist

• Common Threats

• Doxing

• Known Exploits

• Zero Day Exploits

• Social Engineering / Spam

• How Are We Getting Hacked

• Disclaimer

• Telemetry

• What We Can Do

• Which Linux Version

• Pop!_OS

• Choosing The Device

• Which Processor

• Installing Pop!_OS

• Configuring Pop!_OS

• Your First Ride In The Terminal

• Next Issue

Recap

Last week we've had a look at how to set a risk-target at the portfolio level and how to scale our position sizes to meet this target based on a continuous forecast created by an EMA8-32 crossover.

If you didn't yet, you can check out the article here

The plan was to follow this up with a backtest demonstration. However, something unexpected has come up, which requires most of my attention right now. That's why we're momentarily going to step away from the discussion and instead have a look at a substitute article about Operations Security 101, which has been in draft mode for some time now. This is a somewhat longer article so you might not be able to view it in full length within your mail application. Click here to view it in your browser instead.

What Is OpSec?

Operations Security - short OpSec - can mean lots of different things depending on the context of the "operation" you're looking at. In a more general sense, OpSec describes the process of identifying whether your actions and information can be observed or obtained by external actors, which can use this information to exploit or hack your "operation".

But OpSec doesn't stop there! Identifying potential attack vectors is only one part of securing your business. The next, at least equally important part, is the selection and execution of measurements to mitigate the possibilities of getting exploited or leaking critical information in the first place.

Unfortunately, even though everybody knows about Maslow's Pyramid of Needs, having worked as Penetration Tester and CISO, I can confidently say that almost nobody takes this as serious as they should. This is especially sad since most actions you as an individual can take are pretty straightforward and offer high degrees of security.

100% Security Doesn't Exist

Technically anybody could get hacked at any point in time. There's no such thing as a 100% secure system. And depending on what threat you're up against, you really don't stand a chance. If NKs Lazarazus group somehow decides to target you with all their financial backing, knowledge and manpower, you mustn't slip up at any second to not get owned! Which is harder than one might think.

Luckily it makes zero sense for them to target you if you're not a high profile target with insane amounts of access, knowledge or funds. We're talking something like government level access or millions of dollars.

The threats you are facing are not even remotely comparable!

This doesn't mean you should ignore security entirely. Even though we can't get 100% security, we can come remarkably close by utilizing proper techniques and most important of all discipline.

Common Threats

To get a better idea of what proper techniques might look like, we can take a look at some of the most common attacks the average internet user might face:

Doxing

Doxing is commonly referred to the act of publishing sensitive details about a person online, including full name, home address, telephone numbers, family members, date of birth, social security number, employment details, etc. This information can then be used by others to spam your inboxes, steal your credentials or even visit you in real life.

Known Exploits

Known exploits are verified vulnerabilities which can be found in outdated software. For example if you open PDF files (like pirated ebooks) in a vulnerable Adobe application you could be unintentionally installing trojans that were embedded in them, setting up a backdoor for intruders.

It is a must to ensure the software you use is always up to date!

It's also a good idea to realize that nothing's ever really free. If someone is sharing pirated stuff with you, they might have other intentions than just gifting things to random strangers on the internet.

Never download or open a document from any source which you do not fully trust! And even if you trust the source, it's better to have dedicated old burner laptop as your "warez" device and never connect it to the internet.

I have an old laptop which sole purpose is to launch some weird proprietary legacy 3rd party application, that's somehow a heads up sit n go poker video course. When I launch it, all kinds of trojan alerts go off. I verified like 90% of them to be false positives and chances are the rest of it are too. But keeping it on a non-primary offline device is about the best protection you can get without any additional effort. The device can only be accessed physically, not remotely by some trojan or virus.

Zero Day Exploits

Zero Day exploits are vulnerabilities that aren't widely known yet but are getting actively exploited in the wild by threat actors. There's not really much you can do about it other than staying disciplined during internet usage (more on that later) and praying that the softwares developers fix them ASAP.

Social Engineering / Spam

Often times, when someone got "hacked", they didn't really got "hacked". Makes sense? Probably not. Let me clarify that..

Consensus about getting hacked is some notion like: you, as the victim, didn't do anything but some attacker gained access to your system via lots of hard work sitting in a dark room, typing cryptic symbols into his laptop for hours on end.

More often than not, this is not the case though. Most people get "hacked" because they clicked some damn link in an email, installed some questionable program, ran a custom executable that got sent to them or plugged an unknown device into their computer, which then opened a backdoor in their system.

All of these come in basically two forms:

One of which is spam. Each and every existing mailaddress is getting bombarded with spam emails that promise either some riches or sexy dates, which you ought to claim by clicking on a link. Most mail providers do a pretty good job at filtering those out but some get through nontheless.

Another, more sophisticated version is when you're getting approached, either online or in real life, with some promising offer tailored towards you or your hobbies. Again, this can be some kind of enticing offer, that's especially enticing to you because it fits within your area of interest. It could also be some kind of blackmail attempt, claiming to have recorded you while watching porn or other things.

In almost all cases, it's best to just delete and ignore these entirely. Again, nobody really wants to give you something for free.

How Are We Getting Hacked

So what can we do to reduce the chances of getting exploited? To come up with ideas on how to harden our security, we can take some inspiration from what bad actors might be doing to attack us. Hacking a system doesn't work like hollywood makes it look like:

It has a well defined process that starts with something called discovery or reconnaissance. During this stage hackers try to gather as much information as possible about the system they want to target. Now "system" can be a lot of things: your computer or any of your devices, you as a person, your online accounts, etc.

No matter the system, at this stage the attacker uses a mix of passive and active scanning and enumeration techniques. One of such techniques is called Open-source Intelligence, which describes the "collection and analysis of data gathered from open sources to produce actionable intelligence." (Wikipedia)

The next step is a research stage in which attackers dig deeper and research any information they found on the system for known vulnerabilities or possible attack vectors. During the first cycle of an active engagement (aka hack), attackers usually look for low hanging fruits like known remote-code execution in outdated software, leaked passwords or easy ways to slip you malicious links. Anything that can establish a quick and easy tunnel into your system really.

After that comes the phase that everybody knows from films: the exploitation phase. Attackers start cracking hashes gathered via OSINT, attack your online accounts with credential stuffing, try to get you to click links or start actively exploiting running applications like websites you own, etc. If they don't succeed, they will circle back to stage 1 - Reconaissance - and do it all over again. Until they found a way in or lose interest.

After setting foot in your system, hackers start another reconaissance process but this time from the inside. Instead of passively scanning outside resources they are actively enumerating your whole network. Map out how your system is connected to other systems they then try to pivot into, escalating their privileges from "normal user" to administrator access where possible until they infiltrated the whole network.

(Ignore the Reporting Tab in the graphic. This is only relevant for white-hats that are penetration testing systems to then report their findings to the systems owner. Not so much for malicious hackers)

How Can We Protect Ourselves

Unfortunately, we can't talk about how to mitigate all of this in this blogpost. There's so much to say that it probably won't even fit a normal-sized book.

What we can do however, is taking inspiration from this process and target low hanging fruits first to increase our security in a big way with small effort. And then gradually increase our protection as we go. In the future we're probably going to release an OpSec related article every other week or so.

Security is a marathon, not a sprint! Every step taken helps. No matter how big of a step.

The remainder of this article will focus on the OSINT process, giving some rough guidelines on what we can do to protect critical information, and then presenting the most effective action you can take right now: changing your operating system to linux.

Establishing a secure and private computer is a priority before tackling any other task. It can become impossible to achieve security if someone is capturing your screen or sits in your network, intercepting traffic and communication. We need a clean and secure computer untouched by anyone!

Disclaimer

The exact steps taken during the writing of this may need to be modified in order to match updated software and services. Use the overall methods as a guide and not the exact steps. I encourage you to confirm all of my suggestions online before execution. There may be better ways of doing things today. Next, there is no perfect playbook for everyone. You do not need to replicate every step I take on behalf of myself. You may identify a better privacy plan for yourself than the specific examples presented here. And as always things written are not any type of advice. They are merely explicit examples of the actions I have taken. There will be a balance of enjoyable living and refusal to submit to the standard abuses of data collection.

Telemetry

So what can we do to deny attackers doing OSINT to gather information about us? To come up with a plan we first need to understand how information about us gets out into the internet in the first place.

Telemetry is a concept utilized by almost all companies, which defines the act of collecting, storing and processing sensitive data about you and your (computer) habits with the purpose of providing you a better experience, tailoring the system towards you.

While the intent might be true and not malicious, from my point of view it poses a huge privacy invasion. To make things worse, corporations are notoriously bad at securing information. Data breaches or leaks happen often, exposing this sensitive information to everyone on the internet.

Here's a somewhat more recent example of how even governments fail to accurately secure your data. You can't even withhold that information because it's the government, that's collecting it.

Link to article

If you want to play around with this idea a little yourself, type your email address into

https://haveibeenpwned.com/

to see which company also failed to protect your data.

But it doesn't even have to go this far. Data mining companies are collecting and scraping all sorts of personal information, matching it to your identity, your social network and your family and then package everything up to sell it as advertising profiles.

Here's an example of what Apple stores about their users, even if they try to stay anonymous by using an alias (and other privacy measures), you can easily confirm this yourself by requesting what Apple stores about you:

• Full alias name and email address provided during account creation

• Alias physical address provided during account creation

• The serial numbers for all devices

• The dates you first used the email addresses with Apple

• Multiple IP addresses possessed during use of the devices

• The internal computer names assigned to all devices

• The dates/times of any reformatting of the systems

• The dates/times and IP address of last access to iTunes, FaceTime, and iCloud

• Your time zones during usage of the devices

• The telephone number provided during Apple account logins

• Songs you listened to through the official Apple Music application

• The moments within the songs when you paused the playback

• Your IP addresses during media streaming from Apple's servers

• Your preferred musical artists identified during their onboarding process

• The serial number of your iPhones (or other phones)

• All podcasts subscribed to through iOS devices

• Titles of podcast episodes which had been completed or paused

• Dates of podcast subscriptions and listening times

• Podcasts which possessed reviews from you, including full review text

• All app purchases, including free apps, downloaded to the device

• All IP addresses assigned during downloads

• All books downloaded through Apple Books

• All IP addresses used during connections

• An export of all entries from Apple Calendar

• Documents and contacts remaining in iCloud

• Auto-stored contacts from Apple Mail

• Recipient email addresses accessed within Apple Mail

• Dates and times of outgoing email

• Real name extracted from outgoing email headers

Or Some excerpt of what Microsoft likes to store:

• Typing diagnostic data from your keyboard

• Microphone transmissions

• Index of all media files on your computer

• Webcam data

• Browsing history

• Search history

• Location activity

• Health activity collected by HealthVault, Microsoft Band, and other trackers

• Privacy settings across the Microsoft application ecosystem This data would make it very easy to identify you, your location, and all online activity.

In addition, if you have a microphone active and didn't disable the appropriate privacy settings, you could be sharing audio throughout the day. And this is only the hardware side of things. Imagine what information gets stored to create advertising profiles of you..

All of this data was monitored, collected and stored while the user assumed he was being anonymous.

Google or Amazon aren't any better!

There simply is no anonymity with Closed Source!

What We Can Do

There are multiple different approaches we can utilize to prevent the dangers of leaking sensitive information. They can be either REACTIVE or PROACTIVE. On the reactive side we can make an effort to remove publicly available information. This can become quite a challenge since notoriously everything that was once on the internet stays forever on the internet. It's still worth to do this regularly and we're going to show you how to do this and provide a template formular in a later issue.

On the PROACTIVE side we have things like spreading misinformation, preventing the collection of this data, not providing accurate details about ourselves or even completely relocate and start over assuming a new identity.

Preventing data leakage is a low hanging fruit because it can be done without lots of effort. Techniques include always using different email addresses for services, using different, secure passwords for each service, storing them locally encrypted in a password manager, using multi-factor authentication, using alias names instead of your real one where possible, using a VPN with integrated Ad, Tracker & Malware protection, using DNS blocking services with blacklists for Ads, Tracking & Malware hosts, using browser Add-Ons to block tracking, and much more.

This week we're going to show you how to set up your own Linux device so you can avoid getting scanned and objectified by big closed-source corporations like Apple and Microsoft.

Why Linux

Linux is the obvious answer to this problem because it doesn't do all of this nonsense out of the box. Technically, you could block all the telemetry of Apple and Microsoft but you need to actively maintain that protection after every little update. In a perfect world, you shouldn't really need to take such measures to prevent privacy invasions.

Linux has a lot of advantages:

• it doesn't require an online account

• never ask for name, email, address, phone, credit card, etc.

• doesn't collect telemetry (or if so can easily be disabled)

• security is better because of attack surface. There are way more Windows users around the world so more time is spent on developing malware for windows

• completely free and open source. Anyone can audit the code. Many active communities do so.

• almost anything can be modified and customized as you like with full control

• very lightweight. runs smoothly even on older devices

• since any device possible, there's no vendor lock imposed on you like for MacBooks

Almost any version of linux will be better privacy and security wise than MacOS and Windows.

Which Linux Version

Since linux is open-source there are quite a few versions out there to choose from. Anybody can spin up their own interpretation of how an operating system should look like.

If you've ever heard of QubesOS, it's basically a linux version that encapsulates your whole system with docker-like containerization. You can have different boxes for different tasks so your banking box doesn't know about your internet browsing container and vice versa. It might very well be the most private OS but has a very shallow learning curve due to its complexity.

Easier alternatives include distributions like Debian and Ubuntu and their subversions. For linux beginners the best choice is probably Ubuntu.

It allows easy access to software packages in a graphical interface and has some of the highest compatibility with existing computers. It also provides easy software update options, reflecting a large portion of Linux users with abundant online support. Ubuntu has way fewer driver issues than other systems when adding new hardware.

Pop!_OS

However, we're going to focus on Pop!_OS instead of Ubuntu. It is based on Ubuntu, but redesigned for privacy and security. The installer applies full-disk encryption by default. All telemetry is disabled and third-party connections are opt-in. The app store can install and update Flatpak/Deb programs. Snap is not installed. In addition Pop!_OS has better window tiling options and includes a recovery partition to easily restore your system when needed. Overall it just feels more polished.

Almost all steps in this installation guide can be used with Ubuntu, Mint or any other Debian distribution with minimal, if any, alterations.

Choosing The Device

If you're a macOS user, you're very limited to specific hardware approved and sold by Apple. If you want a laptop you only have a handful of options to choose from.

With linux - since it is so lightweight and can run on almost anything - the options are endless! You probably already possess an old laptop which is capable of running Pop!_OS. One of the best ways to test linux is to install it onto a retired or used machine. I keep old laptops specifically for testing new linux builds.

If you only have one computer, a linux virtual machine presents the ideal option for experimentation. However, this is not focus of todays article. We're going to set it up on a physical device in just a few moments.

A new computer is almost always my preference. Acquiring a new device as your security base mitigates any previous infections or issues your already existing devices might have fallen victim to. You might even be able to specify that you want linux installed on it when purchasing to spare some extra work and time.

I typically don't recommend desktops. Laptops are portable and include a screen, keyboard, and trackpad. If you need a desktop, you know your reasons why and do not need my input.

Which Processor

The only thing that's really important during device choice security wise might be the processor, which is the main chip that allows your computer to function. Virtually every Intel processor made since 2008 possesses the Intel Management Engine (IME). IME is a proprietary firmware embedded directly onto the chip. It is a small operating system which runs while your computer is running, asleep or even during standby.

AMD processors also have this option, which is called AMD Secure Technology (AMD ST). Their purpose is to allow your device being managed within a low-powered state. Network administrators take advantage of this to remotely control many aspects of the device within the network.

Now to be clear, this is different from the telemetry invasion! Concerns only arise due to the overall secretiveness around their closed-source code and several known vulnerabilities reported since 2017. You might never have a problem using an Intel ME enabled chip but you could - and I do - argue that any new zero day exploit for IME can be used by criminals.

So if I ever would recommend a processor, it would be one with IME disabled!

The most active vendor of linux based machines with IME disabled by default is System 76. In addition they also use open-source firmwared called Coreboot which replaces the proprietary code included with most motherboards. Coreboot is typically faster and more secure than other BIOS version.

Installing Pop!_OS

Now that you've - hopefully - opted for an IME disabled and Coreboot device, we're ready to install Pop!_OS on it.

Step 1: Navigate to https://pop.system76.com/ and click Download.
Step 2: Choose the Download 22.04 LTS option and allow the download.

Step 3: Install Balena Etcher from https://etcher.balena.io/, launch it and click Flash from file and select the downloaded iso file
Step 4: Click Select target and choose your USB drive.
Step 5: Click Flash and allow the process to complete.

Note that this will completely erase any chosen USB drive, so be careful. As result you should have a bootable USB stick ready for Pop!_OS installation.

Step 6: Insert this drive into your computer and turn it on. Immediately press the key which presents boot options. This is typically, ESC, F1, F7, F8, F10, or DEL. Once you have your boot options screen loaded, select the USB drive with your Pop!_OS installation.

Step 7: Click Try or Install Pop!_OS, select language, location, keyboard layout. Choose clean install and select your internal drive, then click Erase and Install. Provide your desired computer name and password. Select the default option to encrypt the drive. To increase security, choose a different password for the encryption than you chose for your user.

Step 8: Allow the process to complete and click Restart Device. When the computer rebooted, enter your password to unlock the encrypted disk. Next enter your users password to access your Pop!_OS installation.

After gaining access, you're going to be presented a one-time setup. Choose whatever settings you like. Keep location services disabled! Click Skip to bypass any online accounts and Start Using Pop!_OS.

Configuring Pop!_OS

While Pop!_OS is already more secure than Ubuntu by default I still like to take it a step further. These are configurations I commonly use during the setup of my new devices:

Step 1: Launch settings in the lower-right of the dock, toggle Bluetooth to off.

Step 2: Click Privacy in the left menu and disable Connectivity Checking
Step 3: Click File History & Trash and disable everything.
Step 4: Click Screen and change Blank Screen Delay to a longer period
Step 5: Go back to main screen, click Power and disable Automatic Screen Brightness and Dim Screen
Step 6: Click Automatic Suspend and disable all options.
Step 7: Enable Show Battery Percentage

Your First Ride In The Terminal

We're almost done. Just a few more things we need to take care of, this time using the terminal. The linux terminal is a text-based interface which allows you to interact with linux by entering commands. You can use it to execute tasks and navigate the file system and running programs.

Open a terminal by opening the black icon in the Dock (or use the Windows key and type in terminal).

Update your system: copy paste sudo apt update && sudo apt upgrade -y && sudo flatpak update -y && sudo apt full-upgrade -y && sudo pop-upgrade release upgrade into your terminal and hit enter.

This step might take a few minutes. Yeah, that's right.. minutes! Updating a linux distribution will never look like its Windows counterpart where you have to stare at a blue screen for hours on end and cycle through 20 reboots. There's virtually zero downtime when updating a linux machine. That's why you should run this command daily to always stay up to date.

Remove the unnecessary Ubuntu Pro warnings: copy paste mv /etc/apt/apt.conf.d/20apt-esm-hook.conf /etc/apt/apt.conf.d/20apt-esm-hook.conf.bak into your terminal.

Install antivirus: copy paste sudo apt install -y clamav clamav-daemon into your terminal and hit enter. More on this in a later issue. This step is good enough to get basic protection for now.

Remove your background: copy paste the following 3 commands and hit enter after each one:
gsettings set org.gnome.desktop.background picture-uri '',
gsettings set org.gnome.desktop.background picture-uri-dark '',
gsettings set org.gnome.desktop.background primary-color 'rgb(66, 81, 100)'.

You can change the rgb numbers to create the perfect color for your desktop. https://www.w3schools.com/colors/colors_rgb.asp should help.

Add the option to delete permanently to your explorers context menu (rightclick) instead of moving things to trash: gsettings set org.gnome.nautilus.preferences show-delete-permanently true

Next Issue

Congratulations! You now possess a privacy and security hardened linux device!

This was only the first step though. We're going to use that as a base to build up our stack over time. Right now we only prevented the collecting of sensitive data by Microsoft and Apple.

Next we're going to have a look at how to prevent privacy invasion based on Ads and Trackers. For this we're going to use a custom NextDNS profile and a VPN to block invasive connections at the network level so they don't even hit our device.

On top of that we're going to choose and configure Firefox as our main browser to block even more trackers and then talk about some proper ways to navigate and use the modern internet to stay safe and how to remove publicly available information about ourselves.

So long, stay safe!

• Recap

• Why An Index Is Important

• The Most Common Index: B-tree

• Index Leaf Nnodes

• The B-tree

• Installing dbeaver & connecting to your db

• Searching without index

• The EXPLAIN statement

• Creating the index

• Searching with index

• Joining without index

• The gist of it

Recap

Last issue we utilized Docker and docker-compose to create a research data base and then seeded it with free historical EOD price data for 10k crypto tokens.

If you didn't already, you can check out the article here.

This week we're going to take a look at working with databases as a developer, specifically how to create and use an index.

Why An Index Is Important

If you already did some work with databases you probably have heard something along the lines of "SQL is inherently slow". While this might have been true in the past, it's definitely not true anymore. Still, developers often face performance issues when working with SQL databases.

Why does this happen?

SQL is a programming language that separates what data is needed from how to get it. Let's take a look at a very simple example of fetching all the exchanges you can trade BTC on from our database.

SELECT exchanges FROM coins WHERE symbol = 'BTC'

The above SQL query only specifies what data you're interested in: the value of the column named exchanges which is stored in BTCs row in the table coins. We search for the table by matching its column named symbol. What the query doesn't specify is how the database should retrieve this information!

On the one hand this abstraction is good for developers because we don't have to think all that much about how to pull the data from our data store. We just need to tell it what data we want.

The query planner of the database will itself decide how to access that data by simulating different access techniques, scoring them and then executing the best performing one. If we dont set up our database in a way we can pull data efficiently, the only thing the optimizer can really do is going through all rows sequentially, one by one, until it finds what we need.

Right now we "only" have about 8 million rows of historical EOD spot crypto price data including volume and market cap. "Only" in hyphens because even though 8 million rows are basically nothing for a modern database, you will see that how exactly we're searching and accessing its data already makes quite the difference. Things will only get worse over time as we gradually add more data - new EOD crypto prices, other timeframes, different asset classes to diversify better, etc.

Since we're building this datahub for research, we want to be able to fetch data from it (and also execute all other steps) as fast as possible so we can iterate quickly.

This is where an index comes in! It works just like a phonebook. You only have to skim a few pages instead of the whole book for the data you're looking for after specifying an entry point like the first letter of someones lastname.

Let's take a closer look at how an index works.

The Most Common Index: B-tree

The B-tree (not binary tree!) is the most common SQL index and also happens to be the default in most databases to quickly access data. It supports equality and range querying and also works with sorting.

Since it's the default index in PostgreSQL we can create a B-tree index using

CREATE INDEX idx_coins_symbol on coins(symbol);

The query planner will consider using the B-tree index whenever an indexed column is used for a comparison or any of these operators: BETWEEN, IN, IS NULL, IS NOT NULL and pattern matching anchored to the beginning col LIKE 'foo%' but not col LIKE '%bar'.

Creating an index doesn't change the tables data. It creates a new data structure that acts as a pointer and refers to the table.

Searching in a database index is like searching an ordered phonebook. Finding data in an ordered data set is faster because the sort order determines each entrys position.

Index Leaf Nodes

One of the drawbacks of using an index is that changing data via INSERT, DELETE, UPDATE becomes slower because it needs to update the index order immediately to prevent moving large amounts of data.

For this reason the database combines two different data structures to make accessing data faster while also keeping write operations sane: a doubly linked list and a search tree (the B-tree).

If you think of the B-tree as a literal tree, the doubly linked lists are the leafs hanging from its branches. They serve the purpose of establishing a logical order independent from the physical order in memory. The leafs keep a reference of their physical location, matching it to their logical order within this list.

Each entry knows about the previous and next logical entry. If we would only keep track of the physical order, each write operation would need to move large amounts of data to keep the phyiscal order updated, which is very time consuming.

By inserting new leaf nodes between two existing nodes, updating their logical links, we don't need to touch the physical order at all. We just need to move some pointers.

The B-tree

Since the leaf nodes are stored in an arbitrary order - like shuffling the pages of your phonebook - databases put a second structure on top to quickly find the entries again: a balanced search tree with root and branch nodes.

A branch layer is built up until all the leaf nodes are covered where each branch node stores the biggest values of their respective leaf nodes. On top of that is another layer, that stores the biggest values of their underlying branch nodes. This repeats until all biggest values fit into a single node, the root node.

If we search for the key 57, we traverse the tree in ascending order until a value is greater than or equal to 57. Since the tree is balanced - which just means the distance between each root and leaf node is the same everywhere - this search works almost instantly even on huge data sets.

Let's see it in action.

Installing dbeaver & connecting to our db

We're going to use dbeaver to fiddle around with our database for now. Ultimately we should've created the index in our init.sql file while seeding it. After we're done with taking a closer look at our index in action, we're going to add it to the file so it will get created automatically when we redeploy our datahub utilizing docker.

After installing dbeaver, we need to create a new connection for our database, specifically a PostgreSQL connection (elephant).

Select the elephant and click Next. If you're prompted to download the PostgreSQL driver, select Yes and continue. In the next menu you need to fill out some details for the connection. Fill the missing fields with the contents of your .env file so it matches our example and hit OK.

DB_USER=postgres
DB_PW=password
DB_DB=postgres
DB_PORT=5432

You should now be able to connect to the database with dbeaver after starting the datahub with docker-compose up.

Searching Without Index

Let's pull down the information for a randomly selected coin: LOKA. To do this we need to make use of the WHERE clause. Navigate to the coins table by clicking through the connection: postgres -> Databases -> postgres -> Schemas -> public -> Tables -> coins.

CTRL + Click into the green tablename top left to open the SQL editor

and copy-paste the following SQL query in there before hitting the green play button.

SELECT * FROM coins where symbol ='LOKA';

Sure enough we get some data:

The EXPLAIN Statement

Before the database executes any SQL query, its optimizer creates an execution plan for it. Its exact inner workings are not so important right now. It does so by calculating fictive cost amounts for different types of scans for the task at hand, and then comparing them to find the cheapest one.

PostgreSQL (>=9.2) has 4 types of index access mechanics:

• The SEQ SCAN: Scans the entire table as it is stored on the disk step by step until it finds the correct entry.

• The INDEX SCAN: Performs a B-tree traversal, walking through the leaf nodes to find all matching entries, and fetching the data.

• The INDEX ONLY SCAN: Also performs a B-tree traversal but skips accessing the table because it can find selected columns in the index itself.

• The BITMAP INDEX / HEAP SCAN: Works like an INDEX SCAN but instead of fetching only one tuple-pointer fetches all tuple-pointers, sorts them via in-memory bitmap, and then visits the table tuples in physical order.

If there is no index present, the optimizer often has no choice but using a SEQ SCAN and scan the entire table to find the location of the requested data. If possible, we want to never use a SEQ SCAN.

To see which scan is being used for our SQL query, we can just ask the database by putting an EXPLAIN statement in front:

EXPLAIN SELECT * FROM coins where symbol ='LOKA';

As you can see, we're utilizing a SEQ SCAN to access our data. This is because we don't have an index on the column we're searching in yet.

We can also add the ANALYZE keyword to take an even deeper look into the execution plan:

DISCLAIMER: The ANALYZE option executes the statement to record actual timing and row counts! This is mostly harmless for SELECT statements but it modifies your data when using it for INSERT, UPDATE or DELETE operations. To avoid the risk of accidentally modifying your data, you should enclose it in a transaction and perform a rollback afterwards.

EXPLAIN ANALYZE SELECT * FROM coins where symbol ='LOKA';

We're scanning exactly 10,412 table rows in 0.847 ms to find LOKAs row. Is this bad? Well.. let's find out

Creating The Index

Right now, we're only interested in indexing the columns we're actually searching in. To create our index, we need to execute the following query:

CREATE INDEX idx_coins_symbol ON coins(symbol);

Which indexes the entire column symbol on the table coins and stores it in idx_coins_symbol.

Searching With Index

Let's have a look at the execution plan the database comes up with after we've created the index. Again using the EXPLAIN ANALYZE SELECT * FROM coins where symbol ='LOKA' query.

What changed?

Our execution time went down from 0.847 ms to 0.046 ms. Our query is about 18 times faster after adding the index (0.847 / 0.046 ~ 18.4). This is because the database was able to pinpoint the rows location without scanning the whole table to find it due to the B-tree index.

Now you might say, what's all the fuzz? This whole indexing thing sounds rather complicated and we only sped things up by 18x. Well.. yes, because the coins table has only 10K rows in total. Our SELECT query isn't really a real-world use case yet. It's kept simple to make the concept clear. Let's have a look at some more realistic way of accessing data.

But first remove the index:

DROP INDEX IF EXISTS idx_coins_symbol;

Joining Without Index

A common - yet still not really representative - SQL operation for systematic traders is the fetching of all OHLCV rows for a given coin. For example to render its price history as a candlestick chart. Since we've linked the tables to one another using a FOREIGN KEY on the coin_id of the coins table, we need to utilize something like the JOIN Clause or chaining two SELECTs where we wirst find the coins id and then search for it in the ohlcv table. Using the JOIN clause is almost always superior because it gets executed in one go. The two SELECT statements would make two round-trips to the database, introducing latency issues, which are very nasty to deal with at scale.

SELECT ohlcv.*
FROM ohlcv
JOIN coins ON ohlcv.coin_id = coins.id
WHERE coins.symbol = 'LOKA';

It's execution plan looks like this:

It takes us 427.8 ms to get the data. The optimizer used a bunch of different methods to filter for it but again relied heavily on SEQ SCAN due to a missing index. Let's see how things change when we use an index:

Joining With Index

If we want to create an index for columns we're searching for, what columns do we need to consider? We're kind of accessing two different tables via two different columns here. First we scan the symbol column in the coins table to find the coins id, which we then use to find its corresponding rows in the ohlcv table. So at the very least, we should index the symbol column in the coins table.

CREATE INDEX idx_coins_symbol ON coins(symbol);

This only midly sped up things. We're getting our coin id quicker but as we saw, this was only an improvement of 18x on the 10K rows. The planner is still not sure where to look for in the ohlcv table.

Let's put an index on the coin_id column in the ohlcv table too!

CREATE INDEX idx_ohlcv_coin_id ON ohlcv(coin_id);

The planner is now able to use and INDEX SCAN on both tables, which drastically decreases execution time from 427 ms down to 0.393 ms, a 1000x improvement.

The gist of it

Why do we gain a 1000x improvement all of a sudden, the first example only sped up things by 18x?

Because the first example was just a SELECT on a super tiny table. 10K rows is basically nothing. There's only so much performance improvement to be had for this small amount of data.

If we search in bigger tables or combine multiple tables, things get more spicy. Sifting through 10K rows one by one is a whole different story than doing the same on 10 million rows or more.

For now, every time we want to search in a column, we should put an index on it. As the datahub and its requirements grow, we're going to face different scenarios and will adapt to them by choosing the right indexing and accessing techniques.

To round this week of, simply include both b-trees in the sql seed file.

CREATE INDEX idx_coins_symbol ON coins(symbol);
CREATE INDEX idx_ohlcv_coin_id ON ohlcv(coin_id);

- Hōrōshi バガボンド

Disclaimer: The content and information provided by Vagabond Research, including all other materials, are for educational and informational purposes only and should not be considered financial advice or a recommendation to buy or sell any type of security or investment. Always conduct your own research and consult with a licensed financial professional before making investment decisions. Trading and investing can involve significant risk, and you should understand these risks before making any financial decisions