Having separate processes for code deployment and feature releases helps us move faster and with more safety.

• A deployment is the process of getting code pushed through to production.
• A release is the process of getting features available to users.

Deployment, specially in a distributed, multi-service environment, is a complex operation that has many moving parts, including builds and checks.

Multiple services are usually deployed independently and without a specific order. Deploying services in a set order can be cumbersome and can also add complexity to rollbacks. Deployment is linear by nature: a deployment will take to production all the code changes that were made up until the cut-off commit being deployed.

Features, on the other hand, are created in parallel. Multiple people work on various features that span across services. Ideally features should be available to users only after all the code changes are completed and database migrations and dependencies are deployed. Otherwise you run the risk of broken or half-completed interactions.

We separate these two operations with Feature Flags.

What are feature flags?

A feature flag controls if a feature is available for a given user.

• Feature means any set of changes to services and applications, whether backend or frontend.
• User means any actor that may try to interact with a feature, such as a logged in Resend Dashboard user, an API call, or a visitor to the Resend website.

Feature flags can represent boolean states (”does this user have access to the Broadcast API?”) or enumerable states (”which of these versions of the Broadcast editor this user has access to?”).

Feature flags enable precision rollouts. We can use our feature flag library to check whether the current user should have access to a feature and either run the code (if they have access) or keep the old behavior (if they do not).

While this extra added check makes the code longer, this check is temporary, as we'll discuss shortly.

When to use feature flags?

Feature flags enable release synchronization and help minimize risk. Here are some guidelines of when feature flags should be used:

• Progressive rollout: A change to a single service that will be done through multiple pull requests, and some user-facing experience will be incomplete until the last of these pull requests is deployed.
• Coordinated rollout: A change that spans multiple services and needs coordination between the behaviors across these services.
• Sensitive rollout: A change that impacts highly sensitive flows. In these cases, a staggered rollout plan is a good risk mitigator. For example, think: if releasing this causes an incident, will quickly reverting to the previous behavior be helpful?
• Controlled rollout: Any change that we'd like to release in a controlled manner.

The lifecycle of a feature flag

Create a new flag

We use Posthog as our feature flag tool. We give the flag a descriptive name that makes it clear what the feature does and what its value means. For example:

• is-broadcast-api-enabled: evaluates to true if the user should have access to it, or false if not.
• show-broadcast-editor-version: evaluates to either v1 or v2.

Feature flag inputs can take multiple parameters (e.g., user ID, team ID, etc.). For example, we may want to release features to specific users or all users of specific teams on Resend. We can respectively include the user ID or the team ID as an input when evaluating the feature flag depending on our release intent.

Use the flag in code

In either frontend applications or backend services, we use helper libraries to evaluate feature flags. These libraries declare a variable that holds the value of the feature flag evaluation (for example, true or false).

These feature checks makes our release intent clearer for new behavior while the old version is still in place.

Test the flag

When writing unit tests, we include cases for all possible states of the feature flag. For example, we add test cases for when it evaluates to false and check that the behavior is unchanged, and test cases for when it evaluates to true and check that the new changes are in effect. Our testing libraries can be used to force the evaluation of the feature flag to each state.

Release the feature

Using our feature flag management tool, we add rules that change how the feature flag evaluates. While validating our changes, we add our own users and see if new features behave as expected.

If the feature will be tested by beta users, we add them as well and enable beta testing and then collect feedback before moving forward with the release.

Once we're ready to release for all users, we slowly increase the percentage for the feature while monitoring the new feature and any potential issues until we reach 100% of users.

If we detect issues with the feature, we rollback the release as needed.

Remove the feature flag

Our implementation includes an important principle: feature flags are temporary. Flags should not encode business logic that persists indefinitely — they should only control the release of a feature. Lingering feature flag checks makes code more complex and harder to maintain.

Once the feature is released to 100% of users, we may leave it in place temporarily to allow quick rolling back if needed. Depending on the complexity and risk of the release, we may leave it in place a few days or a few weeks.

Ultimately, after a release is complete and stable, we remove all the checks for that feature flag and update test cases accordingly.

Finally, on the feature flag management tool, we delete the flag.