Are you using AWS’s A/B testing tool, CloudWatch Evidently? If so I have bad news for you: As of Oct 16, 2025 AWS will be dropping support for Evidently.

Let me suggest another tool I have been playing with and loving called Posthog. It is a pretty cool tool which allows you to do quite a bit.

For example for both my main websites, Schematcial and Cloud War Games, I can define a series of Marketing Funnels so I can visualize at which step people are falling out of my customer ladder.

Experiments: It also has Feature Flags which for me aren't that exciting on their own but once you pair it with Posthog’s Expriements it really becomes powerful.

I ran a series of experiments for the Cloud War Games Home Page that tested what headlines and element positions would convert the most. Most of them had no statistical effect except for one. The H2 tag above the final call to action. If that read “What are you waiting for? The next outage?” signup likeness was almost double. I am now testing 10+ variations of that header with that text now as the control group.

For anyone building landing pages I strongly suggest you do the same. If you happen to be using React or NextJS I have some components I wrote that I can share with you that might speed up the implementation of this.

Pricing/Options: I am still on the free tier which includes a ton of features. Additionally if you were inclined to set it up it looks like they have a self hosted option. I might give that a try some time.

Let me know if you want me to do a deeper dive on this one or just post hog in general.

Question For You: What tools, if any, are you using for A/B testing?