Why product analytics is completely different than BI

When explorers meet structured formats

Everyone in data has a unique origin story that brought them into data. At least I never met someone who didn’t have one.

My origin story is a classic one (at least I heard it plenty of times from others as well): I was working as a product manager for a marketplace. And my backlog was basically the management team. They assembled once a week to discuss anything about the business, and a part of it was filling my backlog with their input. So my backlog was a collection of features of competitors (90%) and gut feeling features (10%). No surprise; that sucked a lot.

My answer to this was data. Qualitative and quantitative. This gave me some superpowers in these meetings because I could respond with:

- I can see that only 0.3% of users use this feature, and their avg. revenue is not significant

- Based on interviews, user test sessions, and funnel data, we can see that we lost 78% here

- Sounds like a good idea; we can test it if you like against the current one

These things drastically changed the conversation, and the fun returned to me. (Looking back, it was too easy for me, they were not qualified enough to challenge my approaches and deduction and thereby make them and me better, but that came later).

So my approach to data comes from a product perspective. And this is still what drives anything in data for me. When I get frustrated and think, “why am I doing this” - because I want to get good products so that they can provide real progress to the users. You know, the bicycle of the mind thing really is my driving force. Building products that customers love (Marty Cagan had an immense influence on my early work):

Product analytics as a category came a lot later, but I was home there already (even when it was not called that). I was an early adopter of Kissmetrics (omg - I loved this tool so much),

And Mixpanel and Amplitude. Just because Google Analytics never did the job for me. I needed funnels (working funnels), cohorts, and user explorers.

At that time, I did not really realize how different my way of working with data was, compared to other web analysts. Their focus was more on classic marketing analytics or website usage use cases.

And funnily, the same thing happened later when I struggled to find a proper data model for product event data in a data warehouse. And people did not understand why a star schema was not really working for me (40 fact tables for each event type - join hells, you get it). Because their focus was on classic BI and marketing reporting.

And finally, I had so many conversations during the last 12 months about what makes product analytics so hard for product teams and why so few really embrace it and use it in their daily work. Which still keeps me up at night.

That got me thinking - what makes product analytics so different from marketing analytics or classic BI?

The source data

Marketing analytics doesn’t need plenty of source data:

- event data for the core funnel events - these are usually 5-8 events - super easy to implement

- aggregated data from the marketing platforms

Credits: Excalidraw

Classic BI reporting can have plenty of sources, but it can be easily modeled into facts and dimensions (potentially using something like a data vault to handle changes in the source data under the hood).

Thoughtspot demo report - quite a classic BI report. Metrics on time periods.

Product analytics has 2-3 types of source data (depending on the business model):

- event data for product usage - can be up to 30-40 different events (or if not designed properly, some people end up with 300-500 different events)

- user dimensional data - which can change pretty often (forget slowly changing dimensional data)

- account dimensional data (if you have an account feature that can have multiple users) - which can change pretty often

Credits: Excalidraw

These are extremely different requirements already for data creation/sourcing or collection. It takes a lot more effort to implement the instrumentation for product analytics.

Working with the data

In marketing analytics, everything is based on campaign performance and optimization. So you usually end up with a simple presentation layer representing the core funnel with some dimensional data about the marketing context (campaign, ads, channel,..).

The real marketing analytics challenge is handling conversion and cost attribution.

But once you have a solid model and a final reporting table, you are done. Some small adjustments over time, but not much. And the people working with the data will mostly work by comparing different dimensional combinations to find over and underperformers. So mostly filter work. Or you save yourself some time and use a tool like Kausa.

BI is not that much different, you just have more and different reports for the different teams. But you usually end up with some reports and give the users the possibility to filter/breakdown by dimensional data.

Product analytics often starts on a blank sheet. You want to find patterns that tell you which kind of product usage leads to a high customer lifetime value. This sounds simple, but in the end, it is a bucket with millions of variants.

So, we could say that product analytics is exploratory, and BI is descriptive. Which sounds harmless but in practice requires a complete different way of working with the data.

How do you work in product analytics?

As I said before, determining which product usage leads to maximum customer lifetime value is the main objective.

When we unpack this we have first product usage. Product usage consists of:

- a sequence of events (results of user or system actions) of a long period of time (weeks, months, years) conducted by one identified user

- properties for these events (dimensional data) at the time of the event

- properties for the user (dimensional data) that can change over time (quite often not slowly)

Only by this do we end up with millions of combinations we would need to analyze (or parameterize if we want a model doing the job for us).

Customer lifetime value in theory is easier. Just the collected amount over time through subscription or one-time revenue. Since you usually work with identified users, it’s easier to calculate this than other business models. But waiting for monthly revenue to add up is often too late. So you need earlier indicators if your product works and these can be hard to find.

In this labyrinth of options, you need to generalize to get indicators about your product performance and starting points where you want to explore patterns.

These general approaches are usually

#1 the core product journey funnel (you sit down and sketch what a successful product usage looks like, then generalize it a lot, so it fits in a funnel with 6-8 steps).

Funnel in Amplitude

#2 2-3 different retention curves (free vs. paid users, different ones for your main product use cases), and then identify which retention rate you want to focus on (2nd-week retention, 2nd-month retention).

Retention in Amplitude

And then you start to explore. You take your baseline retention curve and start to break it down by different dimensional data. Or you combine different criteria to create different cohorts and then compare them in the retention chart.

This whole process takes time and only gets faster and better when you execute it constantly over a long time. It requires pure data senses, a good product understanding, and, ideally, a combination of qualitative data to get new ideas for quantitative exploration.

But exploring is the key here.

This makes the analyst role and mindset completely different from the one of a marketing or BI analyst. You need a lot more patience and time to conduct product analytics.

This also makes the technical requirements so different. There is a reason why there is still dominant usage of product analytics platforms, even when there is a data warehouse and all other reporting happens based on that data. Because it is mostly exploration with event sequences, and dimensional breakdowns, it would need a crazy SQL hacker to achieve that by writing SQL in a reasonable time.

What does that mean for a company?

If you already have a decent BI setup serving different teams, especially the marketing team with performance reports, adding the product team is not simply an extension.

It will take a different approach, different processes, different mindsets (and therefore, most likely different people), and most likely a different tool for this job.

Good luck if you try to tackle it with your existing analytics (like Google Analytics) or BI setup (Tableau, Looker,..). I can already guarantee you that you will fail.

Adopting product analytics is hard but can be extremely rewarding when achieved. You need to invest first in how you approach the implementation and adoption of product analytics.

Look for the right persons to own this process from day one. These are usually people with product backgrounds but with the mindset to waste hours in slicing and dicing product data. The emphasis is on product background - without it, any analytical talent will not help you.

Then start with defining, tracking, and analyzing the core product usage funnel. Then set up your core retention curves. Once this is in place, the real expedition can start.