Optimizely vs. LaunchDarkly: Which is better for A/B testing and feature flagging?

As your release date draws closer, that dry taste and sinking feeling returns. After building out a set of new features for weeks, you begin to dread the last-minute bugs that are bound to emerge, and feel uncertainty about how users will utilize the new feature.

But rolling out new releases needn’t play out like a horror movie. Feature management tools like Optimizely and LaunchDarkly offer granular feature deployment controls and A/B testing to help you evaluate new features for user adoption and bugs without any repercussions. In this article, we present an unbiased evaluation of both solutions.

Feature flagging allows your developers to hold the release of a new feature until it is fully quality tested in the production environment. For example, say you want to enhance your web app with a new in-app purchase feature. Conventionally, your team would write code in a local, testing, staging and beta environments before finally deploying it to a production environment. But this deployment can introduce unintended bugs or performance lags.

Instead, tools like Optimizely and LaunchDarkly allow you to test and refine the new purchase feature with a small subset of real users in a production environment before releasing it to all of your other users. Because you have an understanding of user adoption before release, you can mitigate risks of going live.

Both tools help you control the features different users see without needing to write any additional code. But LaunchDarkly has a wider range of documented SDKs when compared to Optimizely SDKs.

LaunchDarkly has an intuitive interface that allows you to create a feature flag with a unique flag key. It also allows you to publish the feature flags that are available for all of your client-side and mobile SDKs.

Similarly, Optimizely also has a GUI that allows you to create a feature flag with the required variables. It then automatically generates a code snippet in the development environment of your choice. Deploying this code snippet within your website or app allows you to turn that particular feature off or on.

Both tools offer intuitive user interfaces to A/B test new features and understand how the test results impact revenue or engagement metrics. There is very little that separates both of these tools in this regard.

For instance, say you want to make improvements to your website’s checkout process. You would want to A/B test at least two different checkout processes to select the one that leads to more revenue and fewer cart abandonment rates. You can run this experiment using either Optimizely or LaunchDarkly.

Creating an experiment in LaunchDarkly requires the following steps:

1. Create a metric to measure how different flags perform. You can create a primary metric, such as revenue, and also define a secondary metric, such as the number of added items in the cart.
2. Build the experiment by giving it a unique name and defining the test hypothesis.
3. Turn on the feature flag to begin recording data. At any given time, you will be able to run only one experiment per flag.
4. Start an iteration in the environment of your choice. LaunchDarkly automatically starts a new iteration whenever you make any changes to the experiment.

Similarly, here’s what’s required to create an experiment in Optimizely:

1. Define the URL target or page location of where you would like to run your experiment.
2. Create variations that you’d like to test. Use Optimizely’s visual editor or create your HTML5 code.
3. Identify which subset of your audience will see the experiment. The default option is everyone.
4. Add the metrics that you will measure in the experiment.
5. Allocate how traffic will be sent to all of your test variations.
6. Implement the Optimizely code snippet to publish the test.

Both of these tools help you decouple deployment and release. As a result, your engineers can deploy new features in the production environment without making them available to all of your users. 

Your software team already subjects new features to all or most of the below testing stages:

• Unit tests that check individual units of code for logic 
• Integration tests that test how well various software modules work together  
• Performance testing to identify how your software reacts under load, stress and production traffic
• Regression tests that reconfirm that new changes don’t affect existing functionalities negatively 
• Functional tests determine whether each feature performs as expected or not
• Usability tests that validate how easy it is for end-users to use the new features
• Beta testing that tests the new features with a small set of beta users
• Canary testing that limits the release of new features to a small set of live users

It becomes increasingly difficult and costly to roll back code deployments as they pass through each of these test stages. So ideally, you need the newly developed features before moving them to the next stage.

Both LaunchDarkly and Optimizely help you minimize release risks in different ways.

LaunchDarkly allows you to perform unit, integration, beta, and canary tests right from the production environment. Testing in production saves time and money by eliminating pre-production staging. It also creates a culture of CI/CD and improves software delivery timelines. For instance, you can use LaunchDarkly to first mock test individual pieces of code and then test the whole system with end-to-end integration tests before going on to run canary tests with precise targeting rules.

Optimizely takes a more conventional approach to separating the various test environments. It gives you greater control with multi-stage testing using separate sequential environments. These include a local testing environment that is only accessible to developers, a pre-production environment where code is shared with a larger team along with a small set of beta testers, and the final production environment.

Both tools provide custom pricing quotes based on your unique needs. But LaunchDarkly has a more transparent starting and mid-range pricing plan.

LaunchDarkly has three pricing plans:

• A Starter plan at $8.33 per seat per month, when paid annually
• A Pro plan at $16.67 per seat per month, when paid annually
• And an Enterprise plan that is dynamically priced based on your specific needs

Optimizely Feature Experimentationhas two pricing tiers – Accelerate and Scale. Both of these plans are priced dynamically based on your unique needs. More details on these plans can be found here.

When it comes to feature management controls, both of these tools provide similarly robust ways to help you minimize the risks associated with deploying new features and improve customer engagement as well as conversion rates.

If costs are your biggest concern, then you will find LaunchDarkly’s Starter and Pro plans are more cost-effective. However, Optimizely is a strong contender if you happen to run an eCommerce store or also need a content management system.

Regardless of which tool you choose, you can use mParticle to forward real-time customer data from all of your web, mobile, payment, OTT, and social media sources into either Optimizely or LaunchDarkly. Use mParticle’s integrations to stream customer data to Optimizely and LaunchDarkly without any custom engineering work. 

See how it works by exploring the mParticle platform demo here.