Lies, damn lies and metrics

Metrics are essential, and metrics can be deceiving.

If you’re not tracking any kind of metrics about your open source project or your commercial adoption, that is obviously bad. But it can be just as bad to track the wrong metrics, or to misunderstand the metrics you're tracking.

A recent podcast guest was talking about how his startup had moved from a fully open source license to a ‘source available’ license — primarily out of paranoia that the hyper scalers were going to fork their project, launch a cloud service based on it and steal all their customers. Changing from open source to source available did not, from looking at raw growth metrics, seem to have impacted adoption growth at all. Excellent news, right?

Except that not all new users are created equal if your end goal is monetization. What the company discovered was that while absolute growth was unaffected, not being open source was preventing big companies from adoption the project. They didn’t really understand what ‘source available’ meant, and their legal team would tell them to avoid it. This meant that while overall growth was good, growth among the type of users most likely to become customers in the future was clearly hampered by the license change.

They changed back to a fully open source license.

But it’s a good thing they were looking carefully at the metrics rather than just letting rather superficial growth numbers assure them that the strategy was working.

You need to collect metrics, but you also need to interrogate them closely. In an open source context, this can be a huge challenge, because the readily available metrics are mostly vanity metrics — things like stars and download numbers. You want to know how many users are active, how many users have your project in production. You want to know how growth is in users who are part of big companies versus how many are using your project on nights and weekends.