Data Analysis for Product Managers: Avoiding the Most Common Pitfalls

• May 30, 2025

Leona Jasa

The best product managers rely on data to inform decisions, learn from the past and plan ahead. Many opt to use some sort of dashboard as their first port of call. But, while dashboards are useful tools to keep a high-level eye on product performance, they also tend to be built to answer fixed sets of questions. Typical dashboards track numbers like user growth or daily revenue, or standardized breakdowns like installs per region or average revenue per user. These are essentially surface questions, meaning they tell you whats happening on the surface of the product. To ask deeper questions (usually why things happen rather than if they happen) generally requires data processing and analysis. A dedicated analyst can help to answer some of those deeper questions, but its generally good practice for product managers to be able to do some of their own analysis too. Sometimes youre not quite sure what you want to ask, for example. Sometimes the act of digging in the data can lead to questions that you hadnt thought to ask. Doing your own digging will also help you gain familiarity with your players behaviors, game performance drivers, and a more holistic understanding of the services you manage. Where should you begin? Data analysis broadly requires that you have some familiarity in the following areas:

1. A working knowledge of SQL, Python, R or similar tools
2. What data is available
3. How it is collected
4. Common pitfalls of analysis and statistics

Although it depends on what system your company uses to collect and manage data, there are numerous online and book resources available to learn SQL, Python and other data analysis languages. They are broadly no more complicated to understand than any in-depth knowledge of spreadsheets or other data systems, and a few simple lessons that open up a wealth of possibilities. (For more, here is an hour-long video about getting started in SQL) Similarly, knowing what data is available and how it is collected is probably a matter of reading documentation and speaking with the developers who implemented the system for your game (unless of course youre the one who implemented it). Generally speaking most analytics packages are very flexible in the kinds of data they can draw, as long as developers can update both game and analytics packages to listen for the correct events. It never hurts to ask what data is available, or what could be available. Probably the most important areas of analysis, however, are common pitfalls. These are typical sets of assumptions and mistakes that those new to analysis often make, and which lead them to draw the wrong conclusions from their data. Avoiding these pitfalls is what we hope to teach you in this article.

Common Pitfalls

When engaging in data analysis, inexperienced analysts typically make one or more types of mistake, including:

• Selection bias
• Statistical significance
• Correlation/Causation confusion
• Absolute versus Normalized data
• Cannibalization

Lets go through them. To do so well begin with a hypothetical pricing experiment on a starter pack for Game X, meaning an offer shown to new players intended to incentivize them to purchase in-game goods and habituate them to the idea that purchasing is worthwhile. In Game X the starter pack is currently priced at $1.99, and the question we would like to ask is whether a new price point of $1.49 would optimize conversion and revenue. So we run an experiment on two equally sized populations of newer players (cohorts) to see how they behave when shown one of two packs (also known as an A/B test). We run the experiment for two weeks in order to collect a substantial amount of data, and after the experiment is complete, we obtain the following data:

Selection Bias

Wikipedia tells us that selection bias is the bias introduced by the selection of individuals, groups or data for analysis in such a way that proper randomization is not achieved, thereby ensuring that the sample obtained is not representative of the population intended to be analyzed. In other words that your data and results may not be reliable because the method of collecting samples is inadvertently tilted or filtered. This can happen for any number of reasons, such as built in assumptions about how players behave through to faulty methods of data collection. Nevertheless, if present, selection bias can ultimately lead to faulty conclusions. So avoiding selection bias starts with clear thinking to identify key assumptions and remedy them. For example, our experimental data (see above) might seem pretty simple to work with, but in fact there are several questions to clarify before you begin analyzing it. Questions like these are often called selection considerations. Their purpose is to weed out sources of selection bias (and general noisiness in data), usually by more closely defining who is to be included in the experiment and why. Not identifying selection considerations can seriously skew your results. For example in the above experiment you could end up including all non-purchasing players in your cohorts, regardless of other factors, but that wouldnt lead to useful results. They would likely include players who have seen the current starter pack (and perhaps other offers), as well as players who have been playing for a long time and just never bothered to buy anything in the game. Both of these types of player would likely have very different behavior patterns than the newer player whose behavior we want to analyze, and thus muddy any data we collect. So what selection considerations should we ask for this experiment? Some examples include:

1. Has the player previously seen (and presumably, declined) an existing starter pack offer?
2. Are our two player populations truly representative of the overall game population? (i.e. demographics, known past behavior, otherwise typical play patterns)
3. Are our experiment participants free of other behavioral interference? For example are they also participating in other experiments that could impact purchase behavior?

As you can see, even asking these questions leads us to a clearer sense of who belongs in our cohorts. The players we want to test are those non-purchasers who have not seen current or prior offers (including starter packs), who install Game X only after the period of the experiment has begun, and who are not a part of any other tests. That might be much more of a mouthful to define, but specificity helps avoid selection bias. Once we have clearly identified our selection considerations, we can proceed to define selection criteria (essentially the rules by which we define our data gathering) that apply to the data or its collection method. It can be a little tough, but try to make sure that the criteria you identify can be stated definitively as this makes them easier to code. In the case of our Game X experiment, were looking for players who:

1. Are fairly new (perhaps seven days)
2. Have not seen a previous starter pack offer
3. Are not part of another test
4. Are randomly selected across regions, ages and demographics

These criteria should give us reliable cohorts of players from whom we can draw good data.

Statistical Significance

Statisticians rely on probability analysis to determine the likelihood that a conclusion is accurate. The less data you have in a given set of results, the more volatile (how often individual results stray from the average) those results are likely to be. To understand whether this is the case, they often use Bayesian analysis (a technique that has existed since the 1700s). Bayesian analysis calculates the chances that one set of results could outperform another, with the idea that until there is virtually no chance of this happening you cant be confident in your results. In other words it figures out whether you need more data to be sure. The math behind Bayesian analysis can be pretty complicated, but fortunately you dont need to understand it (but if you would like to, heres a great article on the subject). Instead there are many online calculators ( like this one) that can do the heavy lifting for you. All you need to do is plug your numbers in, and see what comes out. Typically what youll be looking for in a Bayesian analysis is a high confidence interval, a high measure of probability that one set of results will outperform another, or more technically that a population parameter will fall between a set of values for a certain proportion of times. Usually that means youll want your confidence intervals to be 95% or higher, as that generally equates to a result that is statistically significant. For example: lets say we want to get an early look at the behavior of our cohorts to see how things are shaping up. We draw some results and see this:

Correlation vs. Causation

Its practically clich to remind people that correlation is not causation, especially when arguing on the Internet. Nevertheless it happens to be true. When taking action on data, its important to understand whether the connection between the variables you are analyzing is a correlation relationship (meaning that the variables clearly have some kind of relationship but dont directly impact each other), or a causation relationship (meaning that a change in one variable directly impacts the other). For example a rise in property crime correlates to poor economic performance in a region, but one does not directly cause the other. Rather, they are both parts of the same economic system, and mutually dependent/related in some way. On the other hand heavy rainfall in the Arizona desert and flooding of dry river beds in the region are two events in a causal relationship. The data is not merely correlated, as scientific observation has clearly determined that the rain causes the flooding (though not the other way around, obviously). Recognizing correlation vs. causation is an important skill for a data-savvy product manager. Much as with our previous discussions on selection bias and statistical significance, its all about being able to see through what you might feel the data is telling you versus what its actually telling you. Human beings often see patterns that arent quite true and correlate facts that dont quite fit, and in our case this means forming beliefs about what we see happening in our games that lead us to bad conclusions.

Absolute vs. Normalized Data

(Note: in this section, we discuss normalization as it applies to statistical analysis. This should not be confused with normalization as it applies to databases. Same word, different thing entirely.) One advantage of our hypothetical experiment is that both of its cohorts are equal in size (117,000). This means we can compare their results directly. However its often not the case that we get such equalized data, and that can lead to mistaken impressions. If, for example, one of our cohorts was only half the size of the other, then its number of purchases might appear significantly lower in a straight comparison (this is called comparing absolute data). Only through using ratios, such as first calculating conversion before comparing, would we see the true difference. Its usually better to adjust data in this fashion before running a further analysis, a process known as normalizing data.

Cannibalization

The fifth and final common pitfall, as we mentioned earlier, is cannibalization. This term describes a consequence that sometimes occurs when the introduction of a new product shifts the spending or behavioral patterns of players from old to new without any net growth gain for a game. Cannibalization can come in many forms, such as an exciting new game mode that steals players from regular modes but does not grow the overall audience, to a new set of consumable items in a game that takes spending away from existing items. In some cases, cannibalization can even reduce the total overall performance of the whole game, even if the new product is wildly successful.