The next evolution of Product Analytics

about product performance

This is the first time I changed a post's title and minor parts. But some days after I published it, I felt I got the initial angle wrong. This is not about sunsetting product analytics (which was the original title) but more about evolution.

When I began this post, I intended to discuss Optimizely's acquisition of NetSpring and explore its implications for the product category. However, I quickly realized this would evolve into a more fundamental examination of product analytics' current state and whether it makes sense as a distinct category.

A year ago, I wrote a more observation, “Leaving Product Analytics.“ In it, I examined various trends within the category and pointed out signs that product analytics as a standalone field was beginning to fade.

Leaving product analytics

This post essentially closes that chapter. Product analytics started with a promise it could never fully deliver on. I'll explain why fulfilling that promise wasn't possible and explore where things might be heading now. After all, the need for product insights hasn't disappeared—it's just evolving.

Optimizely acquires Netspring

It was quite a surprise when I heard that Optimizely acquired Netspring yesterday. Surprisingly, this news didn't make big waves in the data world. But it makes sense - we're talking about a niche event in the product and customer analytics space. I live in this data niche, so this was big news for me.

I don't want to dive too deep into what this acquisition means, mostly because I don't have many details. Instead, I want to use this opportunity to take another look at the product analytics category and maybe finally close this chapter.

Leaving product analytics - part 1

In that previous post, I mentioned that all the vendors who created and defined the product analytics category were already on their way out. Amplitude was moving towards including marketing analytics capabilities, aiming to become a "customer experience platform". Mixpanel was following suit. Heap was acquired by ContentSquare, so they're now a content experience platform.

But a year ago, new platforms like Netspring, Mitzu, Houseware, or Kubit were entering the category with a new paradigm. They worked directly on top of cloud warehouse data that companies already had. This meant you weren't relying on SDK data anymore. You could use your existing data, model it correctly, and put Netspring on top to do classic product analytics use cases like cohort analysis, funnel analysis, etc.

I was pretty optimistic about this move, and it impacted the category. Amplitude and Mixpanel quickly followed suit. Mixpanel introduced enhanced synchronization. Then, this summer, Amplitude announced the public availability of their native Snowflake connector, allowing you to run Amplitude directly on your Snowflake data.

So I thought, "Wow, we have a new paradigm. Things are moving in a new direction."

But now Netspring has been acquired. Does this give us any feedback about what I assumed was a new movement and horizon?

Netspring is one of two companies that have left the cloud-native warehouse product analytics space. Houseware, which came around a bit later than Netspring, also announced it was making its product analytics product free—which essentially means it's giving up on this category.

It's worth noting that these are all very early and young companies. As with any young company, especially in the current economic environment, getting things off the ground is difficult. These new companies in this space were trying to achieve something challenging.

And that's what this post is about - exploring these difficulties and closing the chapter on product analytics.

What makes it so complicated?

Product analytics has always been a tough nut to crack. In my consulting work, I've seen firsthand how challenging this space can be. If I had focused solely on product analytics projects when I started my data and analytics consulting career, I might not have made it past the first year. The demand for such projects has consistently been low over the years. Even with a profile that showcases my ability to bridge product and data (having worked in both fields), the interest in product analytics remained minimal.

The complexity of product analytics stems mainly from its primary target group: product managers. This group is not inherently averse to data or uninterested in analytics. Rather, their role is so multifaceted and resource-intensive that data analysis often takes a back seat.

Let's draw a comparison with marketing professionals. For marketers, campaign analysis is a natural, recurring task in their weekly and monthly workflows. It's an essential step they learn early in their online marketing careers. Transitioning this work to an analytical platform for cross-campaign analysis isn't a giant leap - it's still part of their core responsibilities.

Product managers, on the other hand, face a different reality. They rarely have the luxury of time to analyze their features or overall product performance. And even when they find the time, there's no straightforward approach to product performance analysis.

Try this: Go on LinkedIn and ask people how they measure product performance. You'll get a variety of responses but no clear consensus.

Marketing metrics are comparatively simple: you have a website funnel, campaigns driving traffic to it, and you analyze how traffic from different campaign sources performs within this funnel. Conversion rates along this funnel give you a solid idea of marketing campaign performance.

In the product world, it's far more complex. Users sign up, use various features differently, and eventually convert to a subscription. The myriad paths they can take make it challenging to pinpoint why they subscribe. It's tough to gauge the impact of specific feature developments on conversion rates - and remember, feature development is to product managers what campaigns are to marketers.

So, we're dealing with a complex problem that lacks easy answers, coupled with limited time for product managers to dig into analytics. Add to this a third challenge: the sheer volume of data points. Marketing analytics might require tracking five touchpoints, often derivable from page views. Product analytics, even with a conservative approach, might need 20-30 events tracked. These must be instrumented initially and maintained consistently to ensure high-quality data for meaningful analysis.

This trifecta - complex analyses, limited time, and resource-intensive instrumentation - makes widespread adoption of product analytics an uphill battle.

Why did Amplitude and Mixpanel succeed, though?

After writing the previous paragraph, I had to ask myself: If product analytics is so complex, requiring significant time and good instrumentation, how did companies like Amplitude and Mixpanel succeed in the first place? Good question!

Amplitude is a public company with substantial revenue and a large customer base. I've worked on numerous Amplitude projects as a consultant, so there's definitely market adoption. How is this possible when everything seems so complicated?

In reality, there's always an ideal state and a current state. Product (and, by that, product analytics as well) has experienced a tremendous hype phase over the past decade. I'm probably not the right person to write about why product got into this hype phase. But certain factors contributed to it.

There were numerous writings about the power of product and its success criteria. The lean startup movement, for instance, focused heavily on product development - how to start with a lean product, incrementally improve it, and get constant feedback. This created a hype cycle around the power of product and what it could bring to a business.

Truth be told, when you have digital software, the product obviously plays a crucial role - without it, there would be no business. But the interesting question is: what kind of role does it play? That's not easy to answer.

Returning to the product hype cycle, as more product teams emerged and companies invested in them, there was a natural curiosity to understand how these products actually work.

You don't see anything in a digital product if you don't track it. Unlike old software, where you'd send a DVD to a customer and only get feedback through license renewals, one promise of digitally hosted cloud products or software-as-a-service was earlier feedback loops to determine if the product makes sense.

I think this early eagerness drove people to invest in product analytics - to get some glimpses into product usage. In most cases today, these are still just glimpses. Most product analytics setups I know of provide small insights into how people use the product - how many people log in during a month, for example. Some go further to understand core activities within the product. However, it's still limited due to complexity, time constraints, and instrumentation challenges.

What can explain Amplitude and Mixpanel's success is that product plays a vital role in our digital world today, and there's at least a basic need for people to understand how a product works. This is what Amplitude and Mixpanel deliver to their customers at a minimum.

However, it also creates a big gap between expectations and what the software can actually deliver. I have many conversations with people using product analytics who are disappointed and wondering why they don't get more out of it. Is it really so hard? First, I tell them, yes, it is hard. It takes work.

Because of this, product analytics itself always has a hard time creating a hype cycle. The returns aren't guaranteed. Marketing analytics has an easier job here—a campaign manager can immediately get feedback on campaign performance, creating a value cycle.

This feedback loop was always missing for product analytics - the loop where you can say, "Wow, we got this kind of value back. Therefore, product analytics totally makes sense."

Additionally, the one thing that always helped product analytics was the success stories of companies that successfully implemented it. All the Uber and Airbnb stories showcasing how they used product usage data to build better features - this was the small hype cycle you could get in product analytics. But when people tried to implement it similarly, they failed because, again, it's not easy. These companies had a lot of resources to achieve that.

Amplitude and Mixpanel created the product analytics category (they were app analytics before). What made them successful was that they gave the first glimpses into how people actually use a product. But going beyond that was super hard for them and their users. This sets the scene for why it might be time to leave product analytics behind.

Leaving product analytics - part 2

It's time to move beyond traditional product analytics—the kind we've been using for the past decade—those superficial glimpses into how people use our products. The problem, as I've mentioned before, is that today's product analytics is too granular. It's disconnected from how products deliver value and businesses genuinely operate.

We've been working with the wrong incentives.

When we ask, "How do people use our product?", we're setting ourselves up for misleading answers. The question itself is flawed. Instead, we should be asking:

• "What value do people get from our product?"

• "How quickly can they achieve this value?"

• "How consistently do we deliver this value?"

And to bridge the gap to business metrics:

• "How does the product contribute to our revenue?"

• "How does it generate initial revenue when we have a free plan and the product serves as an acquisition channel?"

• "How does the product contribute to ongoing revenue in a subscription model?"

These questions lead us to a fundamentally different approach to product analytics. They introduce a higher-level perspective on how we analyze things. This approach also better incorporates the multifaceted role that products play in modern software setups.

In a typical subscription-based, web-hosted software model, products serve various functions. With a free plan, the product becomes an acquisition channel - a phase in the customer journey where users experience the product and determine if it solves their problem. If there's a successful match, they decide to pay for it. In this phase, the product takes on a completely different role compared to the retention role later.

Once a customer signs up for a subscription, the product becomes crucial in maintaining the customer relationship. If the product consistently delivers value over time, customers will happily continue paying for it. Customers will switch if it fails to create value or if a competitor offers more value.

The scenario becomes more complex in enterprise setups where product usage isn't directly linked to contract signings. In cases where sales teams negotiate with CIOs for large license deals, product usage may play a secondary role. However, ensuring a positive user experience remains crucial. It can influence decisions within the company to stick with the product and create a ripple effect when users change jobs and recommend the product to new employers.

Deriving value from a product is paramount, but we need to ensure our analytics setups focus on analyzing the product's value chain, not just how people interact with it. This shift in perspective allows us to align our product analytics more closely with business outcomes and customer success.

How do we call this now?

It's clear that product analytics as a term doesn't quite fit anymore. But categories, especially in tech, tend to stick around - people have these terms ingrained when thinking about specific jobs or problems. So, what's the next evolution of product analytics? And what should we call it?

I see different trends in where classic product analytics vendors are heading. Some are moving towards what they call "customer analytics." I toyed with this term for a while, thinking it might be the answer. But I've come to realize that customer analytics falls into the same trap as product analytics - it's too granular.

When we focus on customer-level analysis, tracking every touchpoint over their lifetime, we encounter the same issues as with product analytics. We end up with a wealth of granular data, which is a good foundation, but it's not where our analysis should end. We need to abstract two or three levels, focusing on value journeys and high-level customer paths. If we stay too granular, we miss the insights that truly create better customer experiences.

This is why I'm wary of the term "customer analytics." I fear it won't deliver the results people are after, much like what we see in the current CDP market (hello CDPs).

So what should we call it? It's something simple, but I understand why vendors shy away from it: business analytics—or good old business intelligence, even if it doesn't have the same buzz. Ultimately, any analytics work should analyze business impact.

Business product analytics

When we look at how people derive value from a product, we're always connecting it to lifetime value. We're always making that link to see how it impacts revenue. Even if we're looking at product usage dropping off, we're really looking at its eventual impact on subscription renewals and revenue.

Interestingly, I'm seeing glimpses of this trend, even in classic business analysis. Financial departments are starting to realize that the metrics they've been working with are purely output metrics - things like revenue, which are the result of many preceding factors.

While it's crucial to have an overview of output metrics, they're not the right level for operationalizing insights. You can't walk into a room and simply say, "We need to make more revenue." You need to break it down to an operational level.

This is where business analytics in general is evolving. We'll still have the output layer showing overall business performance, but we're introducing layers beyond that. We're venturing into analyzing product values, customer movement through product stages, product adoption, marketing journeys, account discovery, and more.

In essence, business analytics is expanding. What we used to call product analytics is becoming an essential part of a larger business analytics picture. We're finally in a position to connect the dots between what's happening in product, sales, customer success, and marketing to see the revenue outcomes.

What I'm hoping for is a next generation of business analytics tools based on event data. Why? Working in product analytics, I've seen the power of event data structures. Unlike classic BI data, event data gives you a full sequence of an account or customer's lifetime. This allows us to build analyses that truly understand value generation.

Event data enables datasets that can answer more complex questions, helping us understand our business better. Current product or event analytics tools can help to an extent, but I see a next generation of business analytics tools based on event data that goes beyond this, answering questions we've always wanted to address but never had the right setup for.

This is why I'm concerned about what happened to NetSpring, which I'd consider a next-generation business analytics tool. Whether they were too early or something else didn't work out, I still hope to see other BI tools move in a similar direction.

Finally, let's talk about at least two sentences about AI. If we can create a business analytics setup using metric trees to show product and business mechanics from output to input metrics, underpinned by an event data model with clear semantics, we could enable some fascinating AI applications. These could identify interesting outliers, segmentations, and developments much more effectively than throwing AI at the snapshot data we currently use in BI applications.

There's an exciting future ahead, but I don't think it's called product analytics anymore. Let's see how it unfolds.