How the answers, how much they vary between seeds, and the runtime of pmsims change across commits on main. Each row is a benchmark scenario; each sparkline runs from the oldest benchmarked commit (left) to the newest (right), with a dashed line where the curve engine became the default.
Click a column heading to sort. Click a row to open its charts.
Sample size. The minimum sample size pmsims returns for the scenario (its min_n), as the median over the seeds run. Lower is not better in itself; the answer should be close to the reference.
Reference. The sample size a separate, much slower simulation finds for the same scenario: the criterion is estimated at sample sizes around the answer with 600 replicates each, and the reference is where it crosses the target. It is our best estimate of the right answer.
vs ref. How far the median answer is from the reference, as a percentage. +10% means pmsims asks for 10% more participants than needed.
Instability. How much the answer moves when the same scenario is run again with a different seed: the standard deviation of log(min_n) across seeds, shown as a percentage (±10% means answers typically differ by about 10%). Needs at least two seeds. Lower is better.
Runtime. Median wall-clock time of one search on one CPU core, in the same pinned environment for every commit. Lower is better; the change is a ratio (×0.5 means twice as fast).
Then, Now, Change. The values at the two commits chosen at the top of the page. Change reads from Then to Now: for sample size, the percentage change in the answer, green if it moved closer to the reference and red if further; for instability, how many times steadier or less steady; for runtime, how many times faster or slower. Hover over a change for the numbers behind it.
Every benchmarked commit, oldest first, with what changed since the one before it (from release notes, merges and commit messages). Open a row for the details.