Split: Feature Management and Experimentation

The main objective of our project was to have a dynamic config with Split without doing a deployment. Split helped us by allowing us to call the Split APIs and fetch the configs on the fly.

Working with dynamic configs is very good and was very easy because we have an environment where we can set up staging, prod, and dev. We can have multiple checks based on which field and which checks we want to apply to a particular config. All these options make us more flexible to change or update the config on a dynamic basis.

The dynamic config updates with Split work without requiring any deployment of the service or software. The APIs we used were very fast and we get the response within 10 milliseconds or so. It is very fast to use an API call for all the config fetching from Split.

Split offers multiple environments, and we have so many fields and checks which we can apply. We can specify countries and apply different checks such as if, else, not in, and in. This provides more flexibility in applying the config based on the checks.

We first try in a dev environment while working on a project. Then we do all the experiments with it and finalize it and move it to the staging environment. There we also do a final round of checks to ensure that all the configs we have defined are working as intended. Then finally we go to production, and there also if we want to have any changes, we can do it on the fly.

The environment feature allows us to approve from dev to staging and then finalize the same thing to prod. The multiple checks are really important features.

The environment feature and the workflow have become very smooth because for a small change in the config, we do not have to deploy the software again. We only have to go and change the config in Split. This gives us more flexibility and a dynamic way to handle and manage the prod config. For example, if I am launching something in India first and then want to launch it for other countries, I only have to go to Split and update the config.

The UI of split.io can be improved.

I have used it while I was working on a project for approximately two or three months.

Split is a great way to handle the config based on checks and in a dynamic way if you want to control your software without doing any deployment. Split can help you there. Split is a great way to dynamically handle config without requiring the deployment of the services or software. This is what I primarily associate with Split. I would rate this review a 9 out of 10.

The main use case for Split would be the experimentation and analytics part. For example, we have two different kinds of components or two different types of pages that we need to test to see which page performs better in terms of users clicking on it via a CTA button. We track them via Mixpanel, making Split an actual use case for determining which variant would be successful in terms of CVR increase.

Recently, we have used Split for an experiment that determined if the user can provide his PII details, such as first name, last name, DOB, etc. Variant A involves entering personal details, while variant B involves entering the employee name to check eligibility. We aimed to find out which would increase the CVR: entering personal details or entering employee details, and we ran this experiment using Split.

The best features Split offers include A/B experimentation itself, multivariant experiments, and a limit exposure feature that allows us to manipulate the traffic going through the split. We can decide to send only 10% of the traffic or 100%, depending on our limit exposure design. Additionally, we have a tracking mechanism and live trail monitoring, making these tools really good.

The limit exposure feature in Split decides if the experiment is healthy or not. For example, we enroll the experiment for 5% of users and check the experiment's healthiness from that. We verify if the on variant is working properly and if the off variant is functioning well to ensure that extra users are not affected if something goes wrong. That is the main feature of limit exposure.

I would say Split has impacted us a lot because we use it for gating new features while going into production, ensuring that those features are not rolled out prematurely. Overall, our organization is positive about using Split.

Split can definitely be improved by introducing something like a library, as we are currently making external API calls that contribute to latency and extra costs for our services. A library would have been better than relying on HTTP calls to improve the speed of Split.

I chose 8 out of 10 for Split because, firstly, it can definitely improve itself by removing the HTTP call and using a library, which would really help our team. Additionally, we tend to move towards other issues related to Split due to flickering issues, which we observed because of latency.

I have been using Split for about two years now.

Split is stable.

Split's scalability is quite good, as it can scale up easily, and the exposure is also fine with a nearly 50-50 ratio. So, it's scalable.

We haven't encountered any significant issues with customer support, but we did have one or two minor issues, and they were pretty reachable.

Split itself has many properties like attributes and split impressions, which are really good tools for Mixpanel tracking of the experimentation. I would suggest that this is a really good product for the use case of experimentation.

Split has increased some of our CVR; for example, in a recent pop-up model experiment, our conversion was around 15%. Drastically, once we started the experiment, the on-variant won by increasing the CVR up to 35%. This makes it a really good tool for determining which particular feature would deliver the expected CVR. It has also reduced the time it takes to determine such outcomes; in approximately two weeks, we decide whether a feature should be rolled out permanently or not. Moreover, it has minimized errors as well.

I would definitely suggest that others use Split for experimentation purposes, not just for gating features, as that is the only benefit we have experienced from using Split. I gave Split a rating of 8 out of 10.