Case Study Tag: How to Know if We're Ready for Membership

Newsroom overview

Who They Are
The Compass Experiment is a local news laboratory founded by McClatchy and Google. They publish Mahoning Matters and the Longmont Leader
Location
Youngstown, Ohio
Founded
2019
Monthly unique visitors
190,000

The Compass Experiment, a local news laboratory founded by McClatchy and the Google News Initiative, launched Mahoning Matters in 2019 in Youngstown, Ohio. They launched on the heels of Youngstown’s longtime local newspaper, The Vindicator, shutting down.

Although the Compass Experiment newsrooms receive funding to get off the ground, they have a goal of becoming financially sustainable in the next few years, so as the team conducted its initial audience research, audience revenue was top of mind. 

This case shows how the Compass Experiment conducted focus groups without an audience of its own in order to assess membership viability. It also shows how they changed strategies when the coronavirus pandemic derailed their plans, what they’re hearing from their earliest supporters as they approach their first anniversary, and how they’ve applied those learnings to the launch of their second Compass newsroom in Colorado. 

Why this is important

Audience research can prevent you from making costly mistakes, especially in the pre-launch phase, when you don’t yet know your audience members. Without an audience research sprint, The Mahoning Matters and Compass Experiment teams wouldn’t have had a strong sense of who their audience was, what they wanted, and whether their audience had any interest in financially supporting their mission. 

But it can be hard to recruit audience research participants when you don’t have an audience. The Compass Experiment’s partnered with other local organizations to get the word out – a smart strategy that a newsroom of any size, especially if you’re assessing information needs or designing a product intended to help you reach new audiences.

Early audience research results indicated that Youngstown residents wouldn’t support a publication with a paywall, which pushed the Compass team toward membership as their model. Early audience research also pointed the team toward emphasizing their financial situation to their early readers as a way to prime them for future reader revenue and membership requests. It also gave them a mission: to be both the journalism of protection – uncovering corruption and misdeeds – and of discovery – telling the full story of Youngstown, successes and all. 

What they did

The summer before Mahoning Matters’ October 2019 launch, the Compass Experiment team headed to Youngtown to find out what residents really wanted from a local news organization if they had the opportunity to start from scratch – and whether they would be willing to support it financially.  The city’s only newspaper, The Vindicator, closed in August 2019, and the Compass Experiment team wanted to know how much residents understood how that was linked to the need for audience revenue. They settled on focus groups since they wanted to collect authentic thoughts, feelings and conversations from this new community they were getting to know.  

Courtesy of the Compass Experiment, September 2019

Throughout August, they hosted a series of focus groups at local library branches across  Youngstown. The library helped recruit participants by hanging flyers inside their branches and sharing the information on their social media channels.

Across three sessions, the Compass Experiment was able to talk to 60 Youngstown residents. Abby Reimer, Senior Manager of UX & Strategic Projects at McClatchy, led the focus groups, which she documented in this Medium post.

Reimer focused the focus groups on guiding questions like: What stories need telling in Mahoning County? What information would improve your day-to-day-life? Reimer’s synthesis for the Compass team focused on what residents strongly wanted, what they strongly didn’t want, and what they were most passionate about.

That winter, Compass Experiment General Manager Mandy Jenkins started recruiting to hire someone who’d be in charge of growth and launching a new membership program for the newsroom. They had big plans: exclusive content, live-events at breweries and summer fairs, and swag. 

Then, COVID hit. 

The Compass team decided to revise their plans to launch membership. They realized it was nearly impossible to come up with alternatives to their previous launch plans. Of course, events were out – and exclusive content felt cruel and out-of-mission for Mahoning Matters to start offering right as coronavirus and Black Lives Matter protests were reaching their 2020 peaks. 

Instead, Compass decided to again double down on audience research. This time, with in-person focus groups impossible and an existing newsletter list of readers, they opted for a survey. They sent the survey to 50 people who financially contributed to the newsroom in the past year. They wanted to know: what do you, our early supporters, like the most about what we’re offering so far? What can we improve? See here for that Mahoning Matters Contributor Survey. 

A sample question from Mahoning Matters’ Contributor Survey

The results

The focus group process in August 2019 helped the team determine what, exactly, their new newsroom would cover. Attendees wanted to easily access and understand public resources like job listings, veterans’ resources, and affordable housing access. Overall, they heard from attendees that Mahoning Matters would need to both be the journalism of protection – uncovering corruption and misdeeds and of discovery – telling the full story of Youngstown, successes and all. 

They also heard loud and clear from the focus group participants, essentially, “If your newsroom has a paywall, we are out.” Reimer’s notes showed the majority of participants felt strongly that they local news should be accessible for as many people as possible. This has pointed Compass Experiment team to membership as the best path forward. 

The team received 20 responses to the survey it sent to the site’s first 50 financial supporters. This was less than the team expected for a completion rate, and they planned to run a virtual focus group with survey respondents who opted into one, but only four people indicated interest.

The team was still able to gather some interesting insights, including what drove people to support them: the newsletter, watchdog journalism, and hyperlocal news. Many said they had been Vindicator subscribers. One said, “The Vindicator closing was a shock to me, and I don’t want that to happen, again.” Another said, “We need independent local journalism dedicated to Youngstown.”

This October, the Mahoning Matters team plans on harnessing the excitement around their one-year anniversary to conduct a wider survey and convene another focus group.  This time, they intend to send out the survey to their entire email list to take a temperature check on how satisfied their readers are with their work, one year in. 

What they learned 

It’s hard to conduct audience research without readers. The Compass Experiment team found it difficult to recruit folks to attend their focus groups early on without an existing base of people or readers. The early focus group participants were mostly a mix of people recruited from Jenkins’ blog post in July announcing that the Compass Experiment was coming to Youngstown (Jenkins had included a dummy email list for people to sign up for updates) and folks that the local libraries recruited from hanging flyers. Another thing the team learned: if you want to attract people to your focus groups, serve lunch! 

Especially in the early days of a newsroom, consider creative ways of reaching the community members you seek to serve. Some creative ways include partnering with a local library to help recruit participants, or purchasing the email list of potential partners in community media. to give you a starting point. The Compass Experiment team was able to put their own learning here into practice when they launched their second site, The Longmont Leader, in Longmont, Colorado.

This time around, the team knew they needed help finding an early target audience. Given the statewide lockdown in Colorado, they had to do surveys and small group discussions virtually rather than at fairs, farmer’s markets, or local microbreweries. So, instead of relying on Facebook lead acquisition and Google ads to build the initial audience list, they purchased the digital assets of the Longmont Observer, a nonprofit community news site, including its email list of nearly 1,000 local readers.

They sent a survey to this list of 1,000 people and, in only a few days, received 128 responses in return. This time, enough people opted in for the focus group that they were able to host three virtual focus groups and dig into their early audiences’ thoughts on local news in the area, their information needs, and what they liked (and didn’t like) about living in Longmont.  

Key takeaways and cautionary notes

Be clear with your audiences about your financial situation. Around the same time the Compass Experiment launched a local news site in Youngstown, Ohio, the local newspaper in Youngstown (The Vindicator) was shutting its doors.

The Compass Experiment learned from early focus groups that people in Youngstown, even die-hard news consumers, had no idea the Vindicator was in such a state of distress. The Compass Experiment is making sure that both of their newsrooms are clear to tell people, “here is our financial situation – we have this runway with Google, but it will run out. New advertising money is not coming in. We will need your help.” They plan on continuing to emphasize this messaging with their readers and early contributors as they gear up for an eventual membership launch. 

Other resources 

Local/Regional Startup (less than 5 years old) How to Know if We're Ready for Membership Compass Experiment Conducting Audience Research Small Newsroom (fewer than 10 people)

Newsroom overview

Who They Are
A local news organization that covers public education in Chicago, Colorado, Detroit, Indiana, New York, Newark, Philadelphia, and Tennessee
Location
Multiple locations in the U.S.
Founded
2014
Membership program launched
2018
Monthly unique visitors
743,000
Number of members
1,100+
Percentage of revenue from membership
1 percent

From the beginning Chalkbeat has had a way for readers to financially support their work, but it was a straightforward donations strategy. They first flagged membership as something to consider in 2014, but didn’t have the capacity to pursue a full membership program at the time. 

In the years following, they continued to grow in readership and revenue, but it wasn’t until 2018 that they had the bandwidth to answer key questions about membership: What would it be at Chalkbeat? Why would they do it? And how significant of an investment would it be to launch and maintain?

Most importantly, they needed to identify the organizational goal for membership: what does success look and feel like, and how could we measure it?

Building on what Chalkbeat knew from years of receiving reader donations, Chief Strategy Officer Alison Go spent two months looking across the organization to develop a framework for determining whether membership was worthwhile and identifying milestones that would tell them if they were on the right track. They launched their membership program in November 2018. 

Why this is important

When assessing whether they’re ready for membership, news organizations spend months choosing their tech stack and designing their membership program. But they often miss a critical step before that: getting specific about what it will look like if membership “works” for them. 

Membership is not a brand campaign that you can toggle on and off when you need a revenue boost or have a bit of extra time. It is both a new relationship with your supporters and a product that you need to manage. Setting off on this path without defining success makes it difficult to assess whether membership is having a positive enough effect on your organization to be worth the significant investment it requires. 

Chalkbeat’s effort to distill its membership experiment into four hypotheses that could be tested and measured within a year of launching a membership program is instructive for news organizations trying to come up with their own definition of success. 

What they did

The first thing the Chalkbeat team established was that membership was not going to be a quick experiment that it could try and fail and then wind down. There would be significant staff, technological, marketing and editorial investment, and the commitments made in the course of a membership program would need to be fulfilled well beyond the end of an end-of-year or spring campaign, said Go.

For two months in 2018, Go was focused full-time on answering a key question: what would it look like for Chalkbeat if membership works? And almost as important, they identified what it would like for membership to not work, and the steps they would take to phase it out. 

“It was not anything technical, it wasn’t like a physical thing or resource, it was the confidence that we would at minimum learn something from this,” Go said.

They identified four hypotheses that they could test by launching membership:

  • Acquisition: Framing small donations as “membership” will drive the highest possible revenue from small donors.
  • Retention: The membership program will have the lowest churn of all small-donor programs.
  • Knowledge community: The membership program will be a critical pillar for a robust editorial knowledge community, ultimately improving the journalism itself and increasing our impact on the community.
  • Costs: The overhead to maintain a membership program will be offset by the benefits. 

They then distilled each hypothesis into a question that data could answer, and identified a “learning time frame” for answering the question, using data from previous donation drives as a baseline. 

  • Acquisition: Does this convert better than our previous campaigns? (1 month)
  • Retention: Do people cancel recurring contributions at a slower rate? If yes, what about the membership program causes this change in behavior? Is it the “membership” framing or specific initiatives within the membership? (3 to 12 months)
  • Knowledge community: Have membership initiatives increased our interaction with our community? Has the quality of our stories improved as a result? Has the reach and impact of our stories increased as a result? (1 year)
  • Costs: What is needed to maintain a membership program and how much does it cost, now and at scale? (1 year)

They decided that if membership did not create incremental revenue but “enabled us to create a feedback loop that we weren’t able to have without it, it was valuable.” 

They also identified revenue milestones that would tell them the role membership would play in the larger financial model. For instance, if membership took 50 percent of an employee’s time to administer and maintain, did it actually pay for itself? And if not at the time, at what size audience would it begin to make sense?

The results

Almost two years after launch, they have more than 1,100 members across the U.S., have generated insights into their four hypotheses, and added one more after launch. This bolstered the case for hiring a dedicated person to support membership. 

Acquisition: They did not see a significant spike in member/donor acquisition in the first two months. Membership grew at about the same rates as their newsletters did.

Retention: Member/donor retention started in the high 90th percentile, and it has remained high since. 

Knowledge community: They have found that it’s easier to engage with members than it was to engage with donors because of the ongoing communication with them, but they are struggling to build local knowledge communities because the membership program is run centrally, at the national level. Kary Perez, who runs their membership program today, says: “As our national team grows, we can continue testing what a national-level knowledge community might look like.” 

Costs: “Our [newsletter] subscriber to member conversion numbers are average. Even at average, we know we can do better. The membership program conclusively pays for itself – the ROI isn’t super high, but it pays for itself. …Even at the launch of the membership program, it was clear it was worth the investment,” Perez wrote in an email. Their launch year, they brought in $53,000 from members, not quite hitting their $60,000 goal – but the following year they brought in $80,000, exceeding their new $55,000 goal.

Synergies with other initiatives: Chalkbeat uses the end-of-year membership campaign to bolster major donor giving and vice versa, such as having a major donor offer a match to motivate readers to become members. Plus, the development team can reuse membership assets and content in their outreach to major donors. 

After their successful launch and end-of-year campaign in 2018, they knew they should invest in a grander version of individual donations, and that they needed a more dedicated resource to do that. In 2019 they hired Perez as the senior marketing manager to manage the membership program.

What they learned

Membership has proven to have both revenue and engagement impact for Chalkbeat. That’s why Perez, the organizational point person for membership, sits between the revenue and editorial teams. They’re still trying to understand which is the primary role it plays, she says. The $80,000 membership brought in for 2019-20 is substantial, but that was only 1 percent of their budget due to their foundation and major donor support. For this reason, it might end up being more critical as a mechanism for greater engagement.

Having membership raises questions about inclusivity. Member benefits are inherently exclusive, and Chalkbeat is asking itself whether it’s possible to square special experiences for members with their organizational commitment to equity and inclusion. “The whole point of our model is that everyone should have access to our work. The questions we ask ourselves are: How does any nonprofit have a membership program? How are we meant to be accessible for everyone while also offering member benefits? How can we create a truly inclusive membership program?” Perez says. 

So far, they have decided that they won’t limit access to anything that’s useful for creating educational outcomes, and they’re exploring how they might create pathways to membership that don’t require a monetary exchange. 

Key takeaways and cautionary tales

A membership program needs measurable goals. Setting off on a path to membership without defining success makes it difficult to assess whether membership is having a positive enough effect on your organization to be worth the significant investment it requires. As Go noted, membership isn’t a quick experiment that can be wound down if it doesn’t “work.” Chalkbeat’s effort to distill its membership experiment into four hypotheses that could be tested and measured within a year of launching a membership program is instructive for news organizations trying to come up with their own definition of success.

Other resources

Local/Regional National How to Know if We're Ready for Membership Digital native Single-subject Chalkbeat Adopting a Product Mindset Developing Membership Metrics

Newsroom overview

Who They Are
An independent media startup in Myanmar covering current affairs and business in English and Burmese
Location
Yangon, Myanmar
Founded
2015
Membership program launched
2020
Monthly unique visitors
120,000
Number of members
421 (plus 147 print subscribers grandfathered in)
Percentage of revenue from membership
25 percent

By 2018, media startup Frontier Myanmar knew they needed to diversify their revenue streams to continue resisting government and commercial pressure and maintain editorial independence. But they knew that putting up a paywall and launching a subscription would shut access off to the overwhelming majority of readers in Myanmar. 

They decided a membership program was the right way forward. But visiting other newsrooms in Asia with membership gave them little insight into what a membership program in Myanmar should look like. So they took the question back to their readers instead, identifying their target users, hosting focus groups, and sharing prototypes along the way. 

Why this is important

Frontier Myanmar had no experience with audience research or design thinking when they decided that a membership model was the right next step. Guided by Asian media consultancy Splice Media, they undertook a several-month audience research and product design process. 

Although they received guidance from Splice, Frontier’s team did most of the research themselves. MPP offers this overview of the steps they took to show that a fairly simple audience research process can yield incredibly useful results if you have an engaged audience already and you know what information you need from them. The focus groups challenged several of Frontier’s assumptions about what types of member-only products would resonate, and gave them confidence that their message of paying to keep Frontier’s journalism free for everyone would resonate.

What they did

Frontier Myanmar began the process in spring 2019, after receiving a Google Digital News Innovation grant. They started simple: a landing page announcing that they were building a community to support Frontier, and asking people to sign up for updates if they wanted to learn more.  They promoted it on Twitter, Facebook, with a banner ad on their site, and in their biweekly newsletter to digital magazine subscribers. Six hundred people signed up almost immediately, and by the end of the audience research phase, which took about a month, there were more than 1,000 people on their beta list. 

At the same time, they identified five target users, which they assessed based on two factors: those who relied on Frontier’s continued existence and saw it as a critical player in Myanmar’s transition to a democracy and who would be willing and able to support it financially. They identified diplomats, journalists, NGO workers, academics, and businesses, then used their personal networks to bring in representatives of each for focus groups (one focus group for each persona, usually with about five people). They also did some targeted 1:1 interviews. They did all this over a month. 

The five personas Frontier designed for (Courtesy of Splice Media)

During those conversations, they asked those users “What do you need for your life? What do you need in order to do business?” (See Slides 22-27 in this presentation from Splice Media to see the types of questions they asked.)

They thought they might hear requests for comments or a members-only Slack. They didn’t. Frontier also assumed that roundtables and panels with politicians, academics, and other experts would be a key component, but they heard from participants that existing organizations like the chambers of commerce already did this well, that there were already more events than people could attend, and users didn’t think Frontier would offer a particularly good event product, Digital Editor Clare Hammond said. Frontier still plans to experiment with casual events, featuring a brief Q&A with a Frontier journalist, followed by drinks, but the coronavirus pandemic interrupted those plans after just one such event (which went well).

Their target users wanted two things: help monitoring Burmese-language media and a daily news briefing that would bring the top headlines together for them. At the time their target users relied heavily on Burmese colleagues to tell them what to pay attention to and many organizations spent a significant amount of money on getting local news translated into English. 

Frontier realized that if they took that responsibility on, they could solve a key problem for thousands of entities and individuals, and save them money. A Frontier membership would be cheaper than individual translation services. 

From there, they designed prototypes of two member-only newsletters:

  • A daily current affairs email newsletter that rounds up the top things to know, including government statements, company statements, top headlines, and other bits of information their reporters hear that don’t rise to the level of full stories
  • A daily media monitoring report that translates the top headlines from the six biggest Burmese newspapers, and fully translates a couple top stories 

They sent those to a beta list for free for two months while they worked out the other details, such as pricing and tiers, which they also surveyed the beta list about. Frontier also regularly surveyed the beta newsletter recipients for feedback on the tone, design, and length. 

The results

The process, from the receipt of the GNI grant, allowing them to begin their work with Splice, to the membership launch, took about seven months.

They launched their membership program in January 2020, just before coronavirus began dominating headlines in the region, with the following tiers:

Frontier Myanmar’s membership tiers (Courtesy of Frontier Myanmar)

They also provided the option to join as an individual (1 login), small institution (5 logins per membership), and large institution (20 logins per membership). Most of their members are individual members, but as of July 2020, they had 16 small institution members with 93 logins total, and three large institution members with another 60 logins. 

About 80 percent of their members are expatriates, which Frontier expected and designed for – their membership products are in English, the pricing is comparable to media products in the U.S. and Europe and they can handle payments in credit cards. They have gotten comfortable with the fact that designing elite newsletter products is what will bring in the revenue they need to keep their journalism work free for anyone to access. 

“Frontier is a bridge between local journalism and international reporting on Myanmar. It’s read by a lot of expats, and [the five target users they identified] pretty much is all the expats in Myanmar,” Hammond said, noting that Burmese speakers are able to access for free what expats needed Frontier to package for them. She added that they do receive support from Burmese readers, but for them it’s more about supporting the mission of independent journalism in Myanmar. 

In January they also launched a “Frontier Fridays” newsletter, a free weekly news roundup that anyone can sign up for. Their goal is to give people a taste of what they could get daily if they became a paying member, and to build a relationship with those for whom a daily news briefing is more than they need. They have 3,800 people on that list.

By July, the revenue brought in from membership already exceeded advertising revenue, which had cratered due to coronavirus. 

Frontier is now forming its first sales team, which will focus on pitching institutional memberships and working with the editorial team to design products that they can find sponsors for, as well as more typical advertising responsibilities. They’ll also soon be sending out a six-month survey to newsletter recipients. 

What they learned

You can’t copy someone else’s model. During the planning stages, Frontier founder and publisher Sonny Swe visited Malaysiakini in Malaysia and Rappler in the Philippines, which both have membership programs. But it didn’t do much to help him figure out what membership should look like for Frontier. That only came when the team sat down with their biggest fans, their readers in Yangon. “You can’t just copy and paste someone else’s model. We live in a different country, different landscape, different spending power,” Swe said. “It has to be tailor-made based on our audience, based on this media landscape.”

Membership is a hospitality business. Swe realized early on that customer service is a critical component of success, so they hired a membership manager who had previously worked at one of the foreign chambers of commerce in Yangon.

Engagement matters more than scale. Swe knew this, in theory. But what really drove it home for him was what happened when a daily newspaper in Myanmar with 22 million followers on Facebook launched a membership program 1.5 months before Frontier launched theirs. The core component was asking people to give 3,000 kyat a week (about $2.25 in July 2020) to access previously free video streams. It’s gained little traction. Swe says that showed him that, “No matter how big your audience is, the most important thing is who your real audience is, who is the hardcore follower. We have only a few hundred thousand, but they feel that Frontier is part of their life and their community.”

Don’t assume you know your audience until you’ve talked to them. Premesh Chandran, co-founder of Malaysiakini, hammered this point with Swe. “Do you know your audience?” he kept asking. Swe says he thought he did because he had been writing for them for years, but accepting that this was not the same as knowing their audiences was a key moment for Frontier. 

Design thinking is a muscle. Going through this cycle of audience research and product design has taught Frontier a new way of designing journalism products. They’ll use these skills again and again. They’re considering applying this approach to future journalism products that they expect they can get underwritten, opening up new revenue streams. 

Key takeaways and cautionary tales

There is no copy-and-paste for membership. While there are a few almost universal truths about what motivates people to become members, what that looks like in practice is highly individualized. The audience research process Frontier undertook was a significant up-front investment, but the return on investment has been equally significant. Building in time to co-design with your engaged readers is likely to help you design more desirable products that gain traction more quickly. This is applicable to all kinds of journalism products, not just membership. 

Other resources

Editor’s note: This case study was edited after publication to correct the year that Frontier Myanmar launched its membership program.

National How to Know if We're Ready for Membership Digital native Discovering our Value Proposition Designing our Membership Program Frontier Myanmar Launching Membership