Havenly's biggest acquisition funnel begins with the style survey: a quick, seven question quiz that identifies the user's personal style and prompts them to register for an account. As the existing style survey became outdated, I was tasked with revamping the survey, including its look and feel, style result algorithm, and conversion optimization. I had the support of the Creative Director, who was the designer for the project, as well as the Head of [Interior] Design, who helped curate photos pertaining to certain styles.
In this case study, I'll walk through key steps in my product process.
Note: To comply with my non-disclosure agreement, I have omitted confidential information in this case study. This information in this case study does not necessarily reflect the views of Havenly.
After gathering objectives from key stakeholders and conducting competitive research, I created the next iteration of the style survey. While the original survey asked questions solely dedicated to style, this new version included questions about style, design needs, and lifestyle. My goal was to make users feel like Havenly “gets” them, which would build trust and ultimately lead to package purchases.
In addition to including bolder, more personal questions such as, “How do you like to spend your time at home? and “If you were a piece of home decor, what would you be?”, I also refreshed the style result algorithm. Each style question had various home decor images to choose from, and each of these images were weighted with their corresponding styles. The user’s selections would determine which images appeared in the following questions, thereby narrowing down the user’s top style progressively. I ensured that all 17 main styles were equally represented in the survey.
After receiving positive feedback from user testing the mocks, we put the new style survey into development. Once built, I implemented an A/B test comparing the original style survey to this new iteration. We stopped the test after two days, as it resulted in a dramatic 26.2% decrease in registrations. Although I believed the test needed to run longer to reach statistical significance, we turned it off due to an executive-level request.
With data regarding drop-off rates for each question and hypotheses about what might have caused the decrease in registrations, I set out to design the next iteration of the style survey. Here are some of my hypotheses about V1, and solutions to improve for V2:
For V1, different stakeholders wanted to add in different questions: the marketing team wanted information for retargeting, the analytics team wanted customer data for segmentation, and the board of directors wanted room-specific questions in the survey. The result was a hodgepodge of both fun and serious questions that failed to coexist in a cohesive way.
Get back to basics with this second version. Pare down the survey so that we ask only fun, style-related questions, and include room-specific imagery to appease the board’s request. From there, we can A/B test adding more informational questions back in, one at a time.
Questions that had simple, visual answers (ie. Do you like this moodboard or this moodboard?) performed extremely well with virtually zero drop off rate.
More of these questions! Nix questions with long, wordy answers and incorporate more visual, succinct choices.
Because each question determines the images the user is served in the subsequent question, we may have narrowed down the user’s style too quickly. Anecdotally around the office, not everybody was getting their “true” style.
Simplify the style result algorithm. Instead of immediately limiting the styles the user is served, show all the style options, and allow the user to choose for herself.
Using these solutions, I created V2 of the style survey, which is what currently exists on the site. I A/B tested this second version of the survey with the original version, and the results were flat after one month of testing. Because neither was outperforming the other, we decided to implement V2 and continue to make improvements from there.
In addition to improving drop-off rates from question to question (from an average of 12.32% across questions to an average of 4.86% across questions), the style result algorithm became more accurate. We essentially "dumbed down” the algorithm: after surfacing all 17 styles, we allow the user to choose as many images as they want. These selections determine the images they’re served in the last question — a Pinterest-type question where the user can again, choose as many images as they want. From there, we tally up their top two styles. With this simplified, more inclusive calculation, almost everyone in the office was receiving their “true” style.
Reflections & Future Considerations
Looking back, I’m curious to see if we could have tested V1 in a more lightweight way, instead of building the survey and then scrapping it after it lost an A/B test. I believe that the survey must deliver accurate results in order to be successful and to coerce people to register — so how can we invest as little time and resources as possible, while still delivering a workable survey? The good news is that V2 of the survey re-used many V1 components, so it was a step in the right direction.
We also made the mistake of calling off the first A/B test completely without fully understanding why. Because we changed the entire survey, it was difficult to pinpoint exactly what wasn’t working. Moving forward, I would continue to optimize smaller parts of the survey such as swapping questions back in, and playing around with the wording. Testing one change at a time allows us to be smarter about what we’re building, and to make the most of our resources. In addition, I’m interested to see how Havenly can ask questions that are both fun to answer and useful to the company. The survey is such an amazing vehicle for obtaining direct information about the user (something that not many companies have), and they should definitely use it to their advantage.