Research 102: Using Data for Decision-Making

After publishing Research 101: Tips for Starting Research Processes, I was asked ‘how do you identify what the right data was in order to help make certain decisions’?

Answering this question will lead you into the process of data interpretation and data-informed decision-making and cyclically send you back into the research process to seek out those next layers of detail or uncover answers to new questions.

My best answer for finding the right data for your question comes back to metrics. In this article, we’ll begin with the basics, we’ll consider how to benchmark your metrics and overview some different types of metrics and the types of data collection these might entail.

Once you really get going, I want to paint the picture of a metric tree, a visual way of understanding how your data and measures of success are connected to one another, the research you conduct, the product you build, and your ultimate measures of success as a business or consultant.

In Research 101, we touched on the fact that different methods of research will collect different kinds of data, such as qualitative or quantitative measures. Metrics are also sometimes called ‘success measures’.

While this is not an official distinction, it helps me to think of metrics as my quantitative data, those numbers that I can track or benchmark to know I am making the right decisions. More loosely, I think of ‘success measures’ as those elements which are more qualitative. Even still, there is some grey area here.

Let’s consider an example:

Maybe you conducted a survey. Consider if this article required people to enter their email and download it and then asked people on a Likert scale how satisfied they were with their download.

Even though the answers might be qualitatively-led (very satisfied, neutral), you can track these as actual numbers, assigning a 3 to a neutral answer or a 4 to satisfied allows you to collect a numeric equivalent to a sentiment (e.g. an average number of people answering the survey were 4.2/5 satisfied).

Alternatively, you could collect scores in categories, such as an NPS score, looking at people who answer either ‘dissatisfied or very dissatisfied’ as negative, and those that are either ‘satisfied or very satisfied’ as positive. In this type of thinking, you may just determine that you want to aim for 80% of users that download your resource to have a positive sentiment.

Benchmarking

As a starting point, you may want to benchmark what is considered ‘good’ for different types of metrics. For example, a 20% churn benchmark for a website is considered good. If you’re running a small, bespoke service or product, you may need to aim for more strict metrics to see value versus products that are driven by marketing and widely available.

Remember, it’s almost impossible to please EVERYONE, and that’s typically not the aim. The goal is to be successful enough with specific experiments that you are driving towards your overall success. The areas where you have gaps are opportunities to be curious and learn more.

For example, above, we asked survey participants how satisfied they were after downloading my article. I want most of the people who read or download my article to find it useful. But maybe someone scored me a 1 because they were actually thinking that my download was a metric calculator and were disappointed that it was not.

This gives me the opportunity to consider whether that is a problem others are facing and one that aligns with my goals of the products I create (spoiler alert: in this highly fictitious example, I will not be creating a metric calculator as my next project).

The goal here is to set myself a metric that I think reflects value. This is what helps me understand what data I need to make the next decision. For example, if I have 20 people download my article and complete the survey and 80% of them are generally or very satisfied, I know that by continuing to create this kind of content, I am providing what my users want or need. I might follow-up with the other 20% to understand why they were dissatisfied but this should not stop my current momentum.

However, if I see that only 20% are satisfied, 50% are neutral, and 30% are dissatisfied, this data has not given me confidence that it is worth my time or effort to continue down the path that I am on. Benchmarking a ‘good’ score will help you understand more complex metrics like churn or cost of acquisition but ultimately, you have to look inside yourself and determine what information would give you confidence or the return you need to keep investing (your time, energy, or funds).

How much to measure

As a new consultant, business, or product, I would advise that it’s important that you focus in on one or two things to track.

Your service or product and measure of success can be most simple when your operations are streamlined (can I pay myself this month or can’t I?). Trying to track everything that could be relevant will suck your time and energy and can put you into decision paralysis.

As your team and product grow, success becomes more nuanced and you’ll need to expand what you track and consider the best tools for helping you synthesise this information. For instance, rather than tracking isolated research results in a spreadsheet you may need a tool like Power BI, Pendo, or Google Analytics.

Spreadsheets are where most of us start, and initially, this is ok until it becomes overwhelming. Once you are unable to monitor data regularly and use this in your decision-making, this is when you need to scale up your tool.

Metric Categories

The list of potential metrics is long and varied. Some of the most helpful and prevalent categories are:

• User experience and satisfaction metrics

• Cost and revenue metrics

• Churn or retention metrics

• Adoption or conversion metrics

• Engagement metrics

• Popularity or referral metrics

Within these categories, there will be specific, granular metrics you can track to show you different facets of engagement (e.g. use metrics might be broken down into frequency, length of time, complexity of engagement, or time to value).

❓ Ask yourself: If there is one single way of knowing that I/my business/my product is successful, what will it be?

The answer to this question will point you to the metric category that will best determine success. This is what you should be tracking and using as a lens to your decision-making.

Decisions are driven by many different factors, including cost and effort so being data-informed means that you are actively considering your actions against your measure of success, but that you aren’t blindly following the numbers without engaging sense.

Frequency and proactive indicators

Next, you will need to identify the granularity with which to measure your success.

Frequency refers to how often you will collect this information. For example, if you determine revenue as your main indicator of success, this can be measured weekly, monthly, quarterly, or annually. While it may be reasonable for an established business to say that yearly revenue (also called annual recurring revenue for subscription-based products or ARR) is their key measure of success, consultants may rely on being able to pay their bills monthly and need to measure this on a more granular level.

⚠️ In this example, are you seeing the red flag yet??

Answer: If annual revenue is my main measure of success, I only know that I have succeeded (or more importantly, if I have failed!) once I have completed the year. Does this sound terrifying?

This is because revenue is a lagging metric, meaning it can only be measured after the milestone has already happened, it’s a measure of the current situation.

Alternatively, leading metrics predict future indicators of success, such as trends or precursory actions. It’s tempting to default to lagging metrics because they give us certainty in knowledge that has already happened.

If our aim is annual revenue, we will be measuring this weekly, monthly, and quarterly to make assumptions or benchmarks towards the bigger picture. Better yet, we want to pair our lagging indicator with a leading measure of success.

This is something that should tell us if we are on the right track towards our goal. For instance, if you give a free 7-day trial before asking someone to pay for a subscription, you might measure the activation of new trial members, track how many trial members convert to paid members, and identify engagement levels for trial members that tend to upgrade to a paid account. Now, you can make proactive decisions.

Maybe, you want to increase your marketing, trying to drive more people to become trial users. Maybe, you want to streamline your upgrade process, making it faster and easier for trial members to upgrade or find the right subscription tier. Maybe, you want to build engagement reminders, prompting users on a free trial to engage with a specific feature if they have not been active within 48 hours of starting their trial.

All of these metrics lead to decisions and, like a ladder, flagging key metrics on each stage of the user journey allows you to climb closer towards your ultimate goals (like a ladder).  

While the below metric tree could be expanded in many directions, I have chosen to show one branch of metrics downwards, expanded into different levels or ladder rungs. These are not exhaustive and there could be different variations. For each context, you have to ask yourself: if I knew this information tomorrow, how would it impact my decision-making. If you can’t answer that, it may not be worth tracking.

In my example, I might feel that I need to increase my revenue by trying to ensure I keep my current users as I have a large amount of new users that I can’t seem to keep active and paying for multiple months.

I might then track down my metric tree from number of customers to churn (an inverse metric as paid users leaving is opposite to my goal of revenue) and from there I can look to track and make decisions that will improve several leading metrics, such as:

• Increasing how frequently users are visiting my website,

• How much time they spend per session, and

• Increasing how many users are completing onboarding.

While decision-making is nuanced, benchmarking and identifying your key metrics aligned with your goals should put you on solid footing. You can also reach out for product mentorship to discuss your particular use case and the metrics (or research) that can help you make informed decisions that you’ll feel confident in.