Product Research & Testing
Product research can’t always accurately predict whether a startup will succeed or fail, but that doesn’t mean it should be ignored. Lina, a Senior Product Delivery Manager at Distillery Tech, talks about how to build a successful product design process in order to properly conduct product testing.
Many entrepreneurs dream of a revolutionary tech project, but the truth is, once an idea is born, the real work has only just begun. As a result, we often encounter entrepreneurs who start out by developing a new product with as many features as possible. Then, they ask vendors to evaluate their products. At times like these, we recommend taking a step back and answering a few questions first:
- Who are we creating the product for?
- What problem does the service solve?
- Is the audience ready for this challenge?
Before the start of development, any idea requires verification. In order to not spend your time developing a product in vain, entrepreneurs must remember that the problem they are trying to solve might not be a problem for other people at all. Additionally, some entrepreneurs will often notice a problem and develop a way to solve it. However, that product might not ever find demand, and few people end up willing to pay for it.
For example, say you found the solution for saving people from the annual shutdown of hot water. It sounds like a revolutionary idea in practice. However, people aren’t always willing to pay for an expensive technological solution to a problem they do not remember for most of the year, especially if there is already a simple alternative, like a boiler. Thus, the product fails.
Analyzing Product Ideas
When telling people about a new product or service idea we have, it is natural for humans to exaggerate their true feelings, making it seem like they are in love with our projects, even if they aren’t. Unfortunately, this makes it difficult to sensibly evaluate the idea and understand whether it will meet demand. Therefore, the first step in every new project should be Customer Development (also known as CustDev or user experience research).
The meeting of an idea with a future user can be started even without a prototype. To do this effectively, talk with prospective users. From there, you can begin to gauge if your idea is worth pursuing.
- Consider the sample size of your audience. It is essential to interview different types of people – those who will give you an understanding of the preferences of the mass user (even skeptical ones). If you only talk to a few super-engaged and loyal people (like friends or family), it’s easy to overestimate the value of a product. Such respondents can provide valuable data, but they also create cognitive bias and reduce statistical validity.
- Avoid leading questions. Don’t ask, “Do you like this idea?”. By doing this, respondents (consciously or not) can overvalue and praise the product – after all, they are not required to buy or use anything yet. Additionally, questions that lead the respondent on can significantly distort the data as well (“This problem worries people so much, would you like to buy a product that will solve it?”).
- Don’t ask for an opinion. The purpose of research is not to collect feedback on the idea but to study the user experience. Respondents should not be experts, assess prospects, say how much the product should cost, and generally express an opinion about the idea (it is unlikely to help). It is more important to study experience – how people currently solve the problem you are focused on, how much they pay for this solution, how often they encounter an issue, what consequences they lead to.
Idea validation can be done on your own, even with little resources. Or, if you are willing to spend, you can involve a contractor as an expert and assistant in your research. If the preliminary analysis shows that the problem is relevant for users, it could make sense to start working on the product based on the identified needs.
User experience learning doesn’t end there. It should be an ongoing process. Take, for example, Orix, a Japanese car-sharing provider. One study found that one in eight customers rent a car not for transportation but for sleep, relaxation, or as a storage cell. Of course, this is difficult to predict in advance, but constant market research and user experience can fundamentally change the original idea (or lead to a new one).
Product Hypothesis Testing
The team tests the idea, which is the key concept around which the product is built. Hypotheses, on the other hand, are more immersive in detail and are usually formed around three themes:
- How to quickly and cheaply convey value to the user using the product.
- How the product will be monetized.
- How users will find it.
It is better to work on multiple issues in parallel because all processes within the product are interconnected. Therefore, it is essential to focus on the project’s income, as we often come across companies with no idea how to monetize their products. This can be justified in some cases, but most entrepreneurs should think about ways to earn money from their product at a very early stage. Moreover, this issue is fundamentally important for attracting investors.
After, you can start creating a simplified product and testing hypotheses on real users. There are several popular options:
- Prototyping is the easiest and cheapest option. It provides rather superficial information and is often used as a way to “look over the shoulder at user behavior.”
- Creation of MVP (Minimum Viable Product). In the MVP approach, teams create a minimum version of the product and distribute it to users. This is more expensive and does not guarantee that the product will be successful and pay off.
- RAT (Riskiest Assumption Test) is an alternative to MVP. This approach suggests starting with testing the riskiest parts of the project. This is an opportunity to quickly and cheaply test an idea without building it entirely. Then, if the assumption is confirmed, you can move on to the following hypothesis. If not, you save development resources.
If you can only receive one type of feedback (qualitative) during the interviewing stage, then during the testing of hypotheses, you will have access to a new layer of information – quantitative data. This can be data received through analytics systems, like Google Analytics, Amplitude, Yandex, or Metrica.
During the research phase, good numbers and estimates increase the likelihood of a successful project launch. However, they do not completely eliminate risks. Pay attention to the implementation of the project: how thoughtful the interface is, how stable the service is, what promotion channels should be used. If there are no questions about the implementation, but the service still does not take off, take a step back again and reevaluate. Perhaps mistakes were made in the pre-project study – check the sample and return to the hypotheses.
Market Analysis & Metrics Testing
The chance of reaching your goals on the first attempt is small, so you need to prepare for changes (both detailed and cardinal) immediately. It is worthwhile to determine the key metrics and reference values you will focus on in advance. This can be the percentage of returning users, conversion to targeted actions, the cost of attracting a new client, or the number of unique visitors.
Use quantitative metrics and A/B tests to track changes and absolute values of indicators. It is worth remembering that it is cheaper to change a hypothesis at the start of a project than to remake a finished service. It is also essential to check the adequacy of the reference values: whether they are too high for a given segment, industry, or country. To do this, you can focus on direct and indirect competitors.
Of course, in some cases, the project does not always achieve the desired performance even after several attempts. Sometimes, it just doesn’t attract users or bring the expected profitability. Unfortunately, there is no clear set of rules for when a product should be discarded. The only situation that will really force you to stop the project’s development is a lack of resources. Of course, some projects can live off significant investments in the early years of launch (for example, Uber), but this is more of an exception to the rule than the norm.
The principles of user experience research are suitable for developing both a small idea and a full-scale project with development over several years. Communication with users, collecting data, and tracking metrics are the keys to success, and the main focus is the experience of the people whose problems you are trying to solve.