22 terms covering AB Testing, Experimentation, Statistics, DevOps, Release Engineering, Metrics, Architecture.
A controlled experiment that splits users into two groups, a control and a variant, to measure which version performs better on a chosen metric. Randomized assignment lets you attribute the difference to the change itself.
Blue-green deployment runs two identical production environments and switches all traffic from the old one to the new one at once. If the new version fails, you switch straight back.
A canary release rolls a new version out to a small slice of users first, watches its metrics, then expands if it stays healthy. It limits the blast radius of a bad change.
A category of entity that a feature flag can target, such as a user, organization, or device. Context kinds let you evaluate flags for things other than individual people.
CUPED, or Controlled-experiment Using Pre-Experiment Data, is a variance-reduction technique that uses each user's behavior from before the test to sharpen the estimate of the treatment effect. It lets experiments reach significance with less traffic or in less time.
Releasing a feature to production with its user-facing behavior hidden, so the code runs and can be tested at scale before anyone sees it. It de-risks big changes by separating deploy from release.
A result that flags an issue which is not actually a problem. High false positive rates train users to ignore output.
A feature flag is a conditional switch in code that turns functionality on or off at runtime without a new deploy. It lets teams control who sees a feature and when.
A feature toggle is another name for a feature flag - a runtime switch that enables or disables functionality without redeploying. The two terms are used interchangeably in most teams.
The accumulated cost of stale feature flags that are left in the codebase after a rollout is complete. Like technical debt, it makes the code harder to read and riskier to change.
A metric you monitor during an experiment to make sure a change does not harm something you care about, even if it improves the primary goal. It acts as a safety check rather than a success measure.
A kill switch is a feature flag whose only job is to instantly disable a feature or dependency in production during an incident. Flipping it takes effect in seconds without a deploy or rollback.
The count of unique users who interact with a product in a given month. Feature flag vendors often bill on MAU, so it directly drives cost.
An experiment that tests several variables at once, measuring not just each factor but how their combinations interact. It answers which mix of changes works best rather than which single change wins.
Progressive delivery is the practice of releasing changes gradually to expanding audiences while watching metrics, using feature flags and canaries to control exposure. It extends continuous delivery with release-time control.
The share of users or traffic that a feature flag serves the new variant to. Teams raise it in steps to expose a feature gradually instead of all at once.
A sample ratio mismatch, or SRM, occurs when the actual split of users between test arms differs significantly from the intended split, signaling a bug in assignment or logging. It invalidates the experiment because the groups are no longer comparable.
A statistical method that lets you monitor an experiment continuously and stop as soon as the result is conclusive, without inflating the false-positive rate that repeated peeking normally causes. It trades a slightly higher sample requirement for the freedom to make valid early decisions.
A measure of how likely an observed difference between test arms is real rather than random noise. It is usually expressed as a p-value below a threshold such as 0.05, meaning under a 5 percent chance the result is a fluke.
A targeting rule decides which users a feature flag serves by matching attributes like plan, region, or user ID. It turns a simple on/off flag into precise control over who sees a feature.
A result that correctly identifies a real issue. The true positive rate is the core measure of a tool's usefulness.
A source control practice where developers integrate small changes into a single shared branch, the trunk, many times a day. Feature flags let unfinished work merge safely without long-lived branches.
Platform benchmarks, real pricing data and progressive delivery practice. No spam.
Free. Unsubscribe any time. See our privacy policy.
Submitting without JavaScript works, but you will land on a plain confirmation response.