When 3,951x is not a speedup

On Friday 11 September at 2:00pm in Matterhorn 2, Nima Torabi and Vitalii Akimov present When 3,951x Is Not a Speedup: Measurement Integrity in Database Benchmarking.

The title comes from a real example. A reported 3,951-fold speedup turned out to be comparing storage rows scanned against query latency, two entirely different measurements conflated into one dramatic number. The figure was arithmetically correct. It was also close to meaningless.

That pattern is not rare. The session works through five reporting errors that recur across published benchmarks. Metric conflation is the one behind that headline figure: reporting one measurement as evidence of another. Trivial workload bias and scale substitution are its close relatives, benchmarking workloads too small to represent production and then extrapolating those same small experiments to production anyway. Topology substitution presents single-node results as evidence for distributed systems. And stale measurements publish results that no longer match the software version being evaluated.

For each one, the session offers a verification rule: a practical way to decide whether an impressive number is genuinely meaningful or an artefact of how it was measured.

Why we're making this argument

It might seem odd for a vendor to stand up at the largest independent open source database conference and argue that performance claims should be harder to make. We think it is the right thing to do. Enterprises choosing a migration path are making decisions that run for years and cost millions, and they deserve numbers that survive scrutiny.

That is the same standard we hold ourselves to. The Liberatii Data Platform lets applications move from Oracle to PostgreSQL without rewriting application code, a claim that only means anything if you can verify it under a real workload.

Come and find us

If you're in Amsterdam this week, come to the session or find us on the sponsor floor. We're happy to talk through what an Oracle exit actually looks like, including the parts that are harder than the marketing suggests.

Not attending? Start a conversation with our team.