Measuring things that matter
and staying away from Vanity Metrics
The Snippet is a Weekly Newsletter on Product Management for aspiring product leaders.
Metrics exist because they measure things you care about, and more importantly measure things that you can do something about if needed.
But the thing about meaningful product metrics is this — they are incredibly hard to define. You need to take a good look at your product, the business model, the reason why your product exists, financial objectives, and then define and instrument a set of metrics that help you drive towards those objectives.
But very often Product teams working under the constant pressures to report progress and to make their product look good, start tracking and reporting useless ‘vanity’ metrics.
Take Click-Through Rates (CTR) as an example— it’s an out of the box metric most analytic engines offer and is pretty easy to measure — digital marketers love to report this performance indicator for their marketing pages, product managers talk about it to show idea validation. But what often is not reported alongside the CTR is the ‘quality’ of those clicks.
How many folks that clicked your ad through to the landing page, ultimately became customers? The CTR just by itself is nothing but a vanity metric. It doesn’t really tell anything about how healthy your product actually is.
For product marketing folks another Vanity metric often cited is “number of page views”. No. of pageviews for your products e-commerce page doesn’t really tell anything, does it?
Similarly, for a team that just launched a mobile app on the app store— the vanity metric that is often reported is no. of downloads. the real question is —are people downloading your app actually using them?
Same with marketing collateral downloads from your website. Who cares if you had a ton of free downloads but no actual sales to show for them?
Other such vanity metrics in marketing are social media followers, page views, subscribers, etc.
Look, don’t get me wrong. All the metrics I trashed in this post so far are important things to be cognizant of and to think about. Heck, what’s wrong with creating buzz around no. of downloads if that gets even more people to download the app and try it?
Look, there is nothing wrong with vanity metrics per se —as long as the Product Manager is not distracted or carried away due to them. But this is exactly what tends to happen in an organization where product management function is new or inexperienced —herd behavior sets in quickly. (Yes, I speak from personal experience)
When everybody around is talking about how great these metrics look PMs who are not strong or don’t know what they are doing tend to get sucked in. Instead of bringing the team back to focusing on what matters, even Product Managers start reporting Vanity Metrics— like “Registered Users” for their SaaS product. Or “A/B tests” that are clearly not set up correctly and are confirmation biased from the beginning.
“Bridge” metrics
I coined the term five years ago —mostly for myself and my product team. To remind ourselves that we need to bridge any and all metrics to actual results that mattered for our product. If we couldn’t connect something to results we cared about, we didn’t actively track those metrics.
Always report your vanity metrics (if you must), alongside bridge metrics to see the real picture. As an example, If you report the subscriber/follower counts for your brand’s social media accounts also track revenue movements due to a % increase or decrease in subscriber/follower counts. If there is no direct causation or at least a strong correlation between the two—why bother to measure and report it?
Same with click-through rates (CTR) — You must report the whole journey. In other words, how many people click your ad, visit your page, become a lead, and then convert to customers. Unless your report final conversions — CTRs are meaningless. They are just a measure of how well your ads are doing in getting people to click them. Even ad marketing agencies don’t care about CTRs anymore!
For Product managers, “Registered users” is a big one to shy away from. Instead, record and report how many of these users actually USE the product and how often. (a.k.a Daily Active Users, Monthly Active Users, etc.)
That’s where the rubber meets the road, isn’t it?
Why do we fall for Vanity Metrics?
In sophisticated product companies with experienced and seasoned Product Management functions, Vanity metrics are already shunned. But in organizations with newly setup Product Management functions or less experienced Product Managers, useless metrics tend to flourish. There are three major reasons.
First, defining useful metrics are more difficult than they appear to be. The product management function must put in some thought on what to measure that will really benefit the product. This requires some foresight and prior experience with building products and dealing with issues related to what is really important.
The second reason is agency issues in companies with siloed departments. In such organizations, despite a clear mandate for business performance indicators, departments continue to measure things that matter to them instead of what matters to the business as a whole.
For instance, the ‘Inside Sales’ department (responsible for generating customer quotations) creates a productivity metric measured by the “number of quotations generated per sales representative”.
Do you see what’s the problem with this? That’s the Vanity Metric. What about the quality of these quotations generated? How many of these quotations result in actual purchase orders? Those are the Bridge metrics.
Thirdly, and perhaps the most important reason why we report Vanity metrics is because they are so easy to report! Most vanity metrics are super easy to set up and very less friction. You can set up google analytics in less than 10 mins and start reporting page views!!
On the contrary, the problem with “Bridge metrics” is that not only do they need some forethought from the product manager, but they are also not out of the box and often not easy to set up.
To set up these metrics, the product management function either must have inhouse skills required to set up the tracking pixels or depend upon other departments or even pay for external help.
Say if you wanted to track a lead through the whole journey of them converting to a customer, you must have interconnected tracking pixels setup that will track
a person clicking your ad on google
landing on/viewing your webpage
converting to a ‘warm lead’ in your CRM (e.g. Salesforce),
and then perhaps after a few days returning to the check-out process
and ultimately buying your product
#1,#2 is easy to measure and is out of the box. That’s the CTR.
For #3 - #5, unless the PM is technically savvy —they will have to rely on external resources to instrument pixels that can measure check out and payment success.
This becomes trickier when your product is sold through offline channels. How do you even track a person that clicked your ad, visited your website learned about your product, and drove to the nearby Home Depot to finally purchase it?
Unless you have access to Home Depot’s data how on earth will you track the final leg of the customer journey? And if you can’t measure this final leg —how do you demonstrably connect revenues to marketing dollars spent?
So you see— Setting up Bridge Metrics is not easy. It can be high friction. As a result, if you are a product manager or product marketer with no access to software development resources, or your product Go to Market is a mix of online/offline channels — it can be hard to track the entire customer journey and report the right metrics that make sense. So what do you do instead? You hold on to, and report what you CAN measure. The CTRs, the Sell Sheet downloads, the Number of Subscribers — all in the hope that you will see an uptick in sales corresponding to an upswing in vanity metrics.
But hope is not a strategy. So, I recommend you don’t do that. Measure the things that you really care about getting right, even if they are hard to measure. Urge the organization to invest in instrumenting the right markers, so that you are not flying blind and are maximizing the chances of success.
The Snippet is a Weekly Newsletter on Product Management for aspiring product leaders.
If you want to connect - I am on Twitter! In case you missed previous posts, you can find them here.