After evals, flywheels
Let’s assume you have a few evals for your AI product in place that allow you to get a good idea of how well the underlying (set of) model(s) is doing and attract a few first users. And now what?
Start building data flywheels! Flywheels are mechanisms that allow you to leverage the data you have to improve your model and attract more users. They are the key to scaling your AI product and the mechanism looks like this:
some users -> some data/feedback -> better model -> more users -> more data/feedback -> even better model -> lots of users… (you get the idea)
Where to start?
A few ideas to get you started thinking about flywheels for your AI product:
- focus on gathering simple, explicit (thumbs up/down) user feedback or implicit engagement data (e.g. clicks, reads, watch time).
- make these mechanisms part of your core user experience.
- use feedback to systematically improve your AI models. This can be tricky, potentially requiring a lot of MLOps infrastructure to continually retrain models.
- ensure at all times that user data is handled responsibly and in compliance with privacy regulations. For example, you might not be able to train on data without explicit user consent (which you can ask for). The good thing is that not all flywheels require training on data for unique users, e.g. using feature-based approaches (averaging across users), privacy-preserving technologies, synthetic data resembling actual usage, or even federated learning.
Further reading
Two resources come to mind:
- a recent blog post Data Flywheel Go Brrr: Using Your Users to Build Better Products by Jason Liu
- the book Working Backwards: Insights, Stories, and Secrets from Inside Amazon also has a chapter on flywheels.
