Experiment: Form completion rates of a pop-up vs. embedded quiz
Does a pop-up with a newsletter sign-up form outperform an embedded quiz with the form placed at the end of the quiz?
Introduction
Pop-ups are widely known to be the most effective way to collect email addresses. They typically display after 4 to 5 seconds while site visitors read your content. Despite being highly disruptive with marketers considering them to be rude and disruptive, they still convert roughly 3%. (Source: Campaign Monitor). Sumo states that top-performing pop-ups can even convert up to 10%, but the average conversion rate is about 3%. (Source: Sumo).
In our other lab experiments we have shown that Riddles can convert over 34% of people who view a lead form if a form is placed without a skip option before showing the Riddle results, making Riddles 11 times more effective for lead generation than pop-ups if the stats by Sumo and Campaign Monitor are correct.
We have numbers from our customers stating that their pop-ups only average conversion rates below 2% and we also know from other customers that their in-Riddle lead forms convert more than 60% of the Riddle takers. We have therefore launched a controlled experiment to compare pop-ups with in-Riddle lead forms, using the same topic and the same form layout for both options.
Assumption
When placing a form inside a Riddle, we can exceed the conversion rate of a pop-up with the same form.
Key metrics to capture
- Conversion rate of the pop-up for people who saw the pop-up
- Conversion rate of the in-Riddle form for people who saw the in-Riddle form
In addition, we also tracked the conversion rate of the in-Riddle form for people who landed on the page with the Riddle and those who started the Riddle.
Experiment setup
We used a random topic generator to pick a random topic for an article that served as the landing page for the pop-up and the Riddle topic. The goal was to find a generic topic that is not news-related and does not carry strong emotions, which might lead to wrong conversion rates.
The chosen topic for the article and the Riddle was: “How well can you read body language?”
The lead form asked for a name and email, had no skip option, and the call to action was: “Unlock your understanding of human behavior and sign up for our psychology newsletter!”
We avoided the free offer or other hooks that would influence the conversion. When using forms in pop-ups or Riddles in a live scenario in your campaigns, we do recommend using a stronger CTA. For the purpose of the experiment, however, we needed to be as neutral as possible to measure true interest in the topic itself, assuming that the willingness to sign up for a newsletter without any additional benefits is higher after the reader has been deeply involved in the topic by answering Riddle questions and spending more time on the topic.
Traffic acquisition for pop-up and quiz
We set up a Google Ads campaign to drive traffic to both pages – the content page with the pop up and the page with the Riddle.
The ad setup only differed in one point: the ad copy was slightly different for the Riddle, highlighting the fact that the page content contained a Riddle about the topic.
All other settings like the keywords, bidding strategy, etc. were the same for both ads. Google did flag our ad for the pop-up version as click-bait and issued a warning about degraded ad performance.
The goal was to get at least 100 starts per variant. As we tested multiple variants of the Riddle for future lab experiments, the total starts on the pop-up version ended up being twice as high as for the individual Riddle variants.
Results
Conversion
Our pop-up performed above the averages quoted by Sumo and Campaign Monitor. We had 2129 views which led to 95 leads. The conversion rate for the pop-up was 4.51%.
The Riddle outperformed the pop-up, even though the readers had to first answer Riddle question before seeing the form.
The conversion rate in the Riddle based on form views was 32.83%, slightly below the averages we see across all Riddles on our platform. Even if we looked at form submissions compared to Riddle views (not people who started the Riddle, just those who landed on the page with the Riddle), the Riddle still outperformed the pop up with a 5.60% conversion rate. Our Riddle generated 115 leads from 2055 Riddle views.
Cost per lead
If you are running pure performance marketing campaigns, you might not be interested in other key metrics like time on site or converting organic traffic and purely focus on cost per lead.
We paid an average CPC (cost per click) of €0.05 for traffic to the pop-up and €0.04 for traffic to the Riddle. This is in line with our overall experience that ads mentioning the word “quiz” outperform similar ads without that word. In addition, the click-through rate on our ads for the Riddle-based ad are 22.97% versus just 13.48% for the regular ad. For longer running campaigns, a higher click-through rate has a positive impact on how Google delivers your ads and also on the CPC.
For the pop-up ad we generated 96 leads from 2129 clicks at 0.05€ per click for a total of 106.45€. This ad generated 96 total leads, resulting in a cost per lead of 1.11€.
The ad to the Riddle generated 115 leads from 2055 clicks at 0.04€ per click for a total of 82.20€, equalling a cost per lead of 0.71€ – a saving of 36% per lead.
Conclusion: quiz beats pop-up
Riddles outpeform pop-ups in all relevant metrics. With a lower cost per lead and better ad performance, we recommend replacing your pop-ups with a Riddle.
There are instances where pop-ups might make more sense, for example when you use a pop-up as a page exit event. But even there, you could consider replacing the pop-up content which is typically a form with a Riddle instead to get higher user engagement.
Riddles come with other advantages as well. Mainly, they are not considered annoying, on the contrary, users typically love taking Riddles and also share their Riddle results on social media. Adweek published a statistic saying that Riddles are the most shared content on Facebook (source: AdWeek) – the statistic is from 2016, but it still holds true based on our experience.
When collecting leads with a Riddle you also get more than just the form data. You can combine the form data with the answers given and the Riddle result, allowing you to segment the users (for instance, segment users by result into subject matter experts and novices and follow up with tailored content based on their skill level).
Lastly, Riddles provide you with a higher time on site. In our experiment, users spent 0.14 minutes on the pop-up and 1.25 minutes on the Riddle. A higher time on site helps your Google ranking and also typically improves your earning potential from other ads on the page.
Free trial
Run your own experiment with Riddle. We provide a full-featured free trial with no credit card needed for sign-up. Click here to create a free trial account for our quiz maker on riddle.com
