How do we test our ideas? Written by Joseph Lichterman Last edited September 14, 2020 You won’t know whether an idea works until you actually get it out there in front of members. There’s no substitute for real-world data. But you also don’t want to pour all your resources into something until you know you’re on the right track. The best strategy for launching effective membership products is to test-and-learn. When you want to use a test and learn strategy to develop and launch a product, you may need to create a minimum viable product, or MVP. Launching MVPs is a way of testing your ideas in the real world. MVPs don’t need to be perfect. They are the most basic expression of your idea or product. When developing an MVP, focus on what you think are the most important features. Keep the list very tight and very focused. You can layer on additional features after you’ve tested your concept. A simple MVP is also helpful because you can easily draw links between your actions and observed impacts. In other words, it’ll hopefully be easier for members to understand your MVP and easier for you to interpret how people react to it. We’ve broken out the process of designing an MVP into four stages: set goals, design your MVP, launch/gather data, and assess/iterate. 1. Set Clear Goals The first step in launching an MVP is to set clear outcome-based goals. What do you want your MVP to accomplish and how will you know if you’ve succeeded? Use this section’s ideation and goal-setting sections to help you learn from your audience feedback and then set outcome-based goals. Another way to think about your MVP is that it’s a hypothesis. What information do you need to gather to prove or disprove your hypothesis? For example, when Daily Maverick in South Africa set out to launch its donations MVP, it’s goal was to answer the following questions: Whether people would financially support The Daily Maverick on an ongoing basisWhere on their owned platforms they would find their most engaged readers What membership messaging most resonated How button placements and color schemes affect signups Quick View How Daily Maverick tested its membership assumptions pre-launch Daily Maverick took advantage of a delay with its membership program launch to answer some questions it had about its potential members. 2. Design your MVP A simple MVP is great because you can clearly draw insights between your ideas and audience behavior, and you can design and launch it relatively quickly. In her guide to making more data-informed membership decisions, MPP coach Federica Cherubini offered advice on keeping things simple in order to determine the link between action and audience impact: It’s important to remember to introduce just one variable at a time, so you can understand which strategies are working, and which ones need tweaking. For example, if you’re testing which platform is converting better, make sure the wording on all calls to action is the same, or quite similar (while wording might vary a bit, you’ll want to make sure the tone remains consistent). Once you’ve tested for platform, then you can test what messaging works best, by trying different wording on the same platform. 3. Launch your MVP Once you’ve established how you want to test your MVP, then launch it. Here’s a short and not at all exhaustive list of how to get your MVP in front of your audience members Tools like Google Optimize hook into Google Analytics and allow you to run light tests on your site. This is useful if you want to test an MVP of a new type of call to action, for exampleMost email service providers allow you to run A/B tests within their systems to probe items such as subject lines, sender name, and send time. Social media management platforms allow users to run tests on everything from post text to preview image and more. There are tools like TestFlight and InVision that allow you put early versions of apps and digital experiences into the hands of potential testers The Narwhal used an A/B test to gauge whether potential members would respond better to an invitation to “Become a member” or “Become a Narwhal,” taking the guesswork out of choosing one that would resonate with its audience members. Quick View How The Narwhal adopted a test-and-learn mindset The Narwhal has developed a rhythm of running small tests to optimize every stage of its audience funnel. One of the key things to remember when testing is that you need to allow enough time and have the right testers for a significant result. You want to ensure that the people who look at your MVP match the type of audience member you hope will adopt it. If your target members are college students, don’t exclusively test your MVP on working professionals. If you’re specifically designing something to reach an underrepresented audience, make sure that members of this audience are included in your MVP testing audience. 4. Assess results and iterate Depending on the nature of your test, you’ll likely want it running for at least one sprint, and possibly longer. It’s important to resist the impulse to cut the test short if a few people react poorly. Make an effort to keep any negative feedback in perspective by comparing the feedback to the total number of people who are actually involved in the test. There might be situations – for example, if a test has had a consistent and overall negative impact on your search ranking or your advertising revenue – when the impacts to your business mean you absolutely have to cut a test short. But in general, you will get your best results if you let your test run its course. After you’ve run your first test, you’ll want to assess the results and improve on your MVP. Revisit the goals you set at the start of your test. Did you hit your targets? Were there areas where you fell short? Did anything surprise you? Identify a few small changes or tweaks that you think might help you meet or improve upon your goals, implement them, and measure what happens. The cycle begins again. When DoR in Romania began hosting pop-up newsrooms outside the capital of Bucharest, they created a series of MVPs that grew in scope over the course of a month. Each MVP built on what they learned from the previous one. MVP 1: A series of after-work conversations in their newsroomMVP 2: A full day hosting a stage at a festival outside BucharestMVP 3: A week in the city of Târgu MureșBeta: A series of trips to the region of Vaslui over the course of a few months Your membership offerings will likely change over time, even within a successful program, because your members’ needs and expectations change. By implementing the test and learn cycle, your membership product can grow alongside your members. Quick View How Bridge Michigan tested its way to membership growth They designed a set of targeted experiments at every stage of the audience funnel.