The BBC Sport app is a big player in the sports app field, and gets 3 million weekly signed-in users. As a product team we had an objective to get all active users signed in to the app.
As one of two designers in the team, I led the design of an A/B test with the objective of informing the BBC Sport app sign-in strategy. At the time only 49% of users were signed-in, and the company had set the objective of getting everyone signed-in, in order to better understand the audience.
There wasn't a fully formed sign-in strategy and there was pressure from senior leadership to get remaining 51% of users signed in by the end of the year, so the experiment would be pivotal in influencing the direction.
Qualitative research showed that the majority of users that weren’t signed-in didn’t understand the benefits of being signed-in.
Although a mature product, experimentation was new to the team, so there was learning curve to implement the test in the app.
Through my network of contacts from previous teams, I was able to connect the product manager to BBC iPlayer colleagues that had experience of implementing similar test. This helped us look at how the experiment could be applied to the Sport app.
I planned and facilitated a cross-discipline workshop to identify test candidates, basing the candidates on reusability, feasibility and viability.
I worked heavily with the team's in-house experimentation specialist and product manager to understand how to gain statistical significance in the test.
I joined a BBC Account working group to regularly share updates and receive feedback from other designers and product managers throughout the project.
I recommended implementing an in app survey at points in the users journeys, in an effort to capture qualitative insight to help make sense of the quantitative data.
I led conversations with BBC marketing and legal teams, to green light the use of BBC Sport talent imagery.
Sign-ins peaked at 93% within 10 days, reaching 99% by day 20, indicating sign-in UI effectiveness waned rapidly.
There was clear qualitative feedback in the survey that helped us understand why some users still felt a barrier to signing in, with the majority of users that took part not understanding the benefits of signing-in.
Product were able to create a more informed strategy for implementing sign-in walls in the BBC Sport app, rolling out a progressively aggressive sign-in wall approach that allowed users to dismiss for 3 months, before making sign-in mandatory to use the app.
Having to sign-in to use BBC services had faced mixed response by the audience, and although weekly users initially dropped from 2.8 million, the app now has 3 million active signed-in users.
Sharing the results of the experiment with the working group helped to steer the overall BBC sign-in strategy, with the BBC account team using qualitative data captured by the interception survey to form a pan BBC strategy that helps to sell the benefits of signing-in on all sign-in walls.
Made in the North West, UK