When I joined Gartner in 2019 one of the products I was brought on to support was a pay-per-click lead generation auction system. Gartner had hired a consulting firm to do generative UX research to understand user needs. Users of this product have auctions based on keyword categories, site placements, and geographic target markets. For larger companies, this could mean potentially hundreds of auctions to manage with different costs-per-click and costs-per-lead strategies.

The legacy auction system tried to solve this complexity by managing auction bids as a percentage adjustment of one default bid price. While this attempted to adjust bids easier many of the larger companies with higher spending wanted more granular control of actual costs.

Problem Statement

Higher spend users want granular control of bid prices so they can run a more market-specific targeting so that they can regularly test price adjustments within a set budget.

 

Launch and Results

At the culmination of all of our UX research, my team held several stakeholder presentations to share research analysis and secure buy-in on several product recommendations.

The first release of the product update was completed six months after my in-house team took over the project. 

• While time on task did increase users across all spending levels welcomed the tradeoff.
• Where we saw some limited contraction in auctions we made up for the revenue loss through higher spending in their existing auctions.
• We prepared for an increase in staff support time dedicated to helping users with this learning curve. We prepared contingency plans with stakeholders if this was prolonged or if we saw revenue rapidly decrease as a result of the shift. After 3-4 weeks, call and email volume normalized, and we never needed to revisit our contingency plans.

Post-launch our product had an increase in NPS ratings, from 6.0 to 7.4, that we could attribute directly to our auction product enhancements. Further, the improved UX led to 15%+ revenue growth year over year based on increased spending and auction expansion in new targeting areas. This growth continued into the beginning of the pandemic and helped keep our growth steady despite the global challenges.

UX Research and Testing

Moving to a more granular auction system had several risks that we needed to address through UX improvements as well as user educational components.

• Setting each bid price manually would initially increase user times on tasks within our product.
• Users might exit some secondary market auctions that were revenue-positive.
• Smaller user teams with fewer resources may be negatively impacted.
• We anticipated needed to dedicate more internal staff time to supporting users through this change which would move resources away from sales.

There were also very strong benefits to this move:

• Users were very familiar with this level of auction controls based on competitors in our space. New users often expected this functionality.
• Users could make more informed targeting and budgeting decisions.
• This would expand our product roadmap capabilities to later include bidding automation and goal based bidding strategies.

We ran several rounds of generative research to better understand how different persona types would respond to this shift in their bidding processes. For each round, we brought different stakeholders onto the calls to let them hear the voices of real users and debriefed with them to alleviate assumption-based concerns.

Throughout the testing, we also captured some additional user needs, including the ability to apply bulk updates, bringing estimations closer to bid inputs to remove extra navigational steps, and pacing notices to help users stay on or under budget.

At this point, we identified several user pain points that needed additional testing:

• Granular bid controls would require additional effort and resources for most users. A subset of those users had not identified a need for granular controls.
• To help users make highly informed decisions at each auction point we would need to provide them with much more data than they previously had access to. This could create a new pain point if it leads to analysis paralysis for users who don’t have the experience or resources to manage this data analysis.

We went through several rounds of prototype testing on concepts like bulk bid adjustments, the ability to work outside of our system with third-party tools and upload bid adjustments via CSV, and user flow adjustments. Through UX research we were able to demonstrate that product hypotheses the team had made, such as bulk bidding features, would not solve the actual user needs and ultimately would be a wasteful feature to build.

Launch and Results

At the culmination of all of our UX research, my team held several stakeholder presentations to share research analysis and secure buy-in on several product recommendations.

The first release of the product update was completed six months after my in-house team took over the project. 

• While time on task did increase users across all spending levels welcomed the tradeoff.
• Where we saw some limited contraction in auctions we made up for the revenue loss through higher spending in their existing auctions.
• We prepared for an increase in staff support time dedicated to helping users with this learning curve. We prepared contingency plans with stakeholders if this was prolonged or if we saw revenue rapidly decrease as a result of the shift. After 3-4 weeks, call and email volume normalized, and we never needed to revisit our contingency plans.

Post-launch our product had an increase in NPS ratings, from 6.0 to 7.4, that we could attribute directly to our auction product enhancements. Further, the improved UX led to 15%+ revenue growth year over year based on increased spending and auction expansion in new targeting areas. This growth continued into the beginning of the pandemic and helped keep our growth steady despite the global challenges.