Challenges

As we’ve never attempted gamification, this feature redesign required a large budget for external research and user testing, which was not always available to us. Another obstacle was the de and reprioritization of this project on the product roadmap. Having to continuously drop our efforts and try to pick up where we left off was a less organized process than would otherwise be the case.

Approach

Seeing that current feature usability was fundamentally challenged, we took various behavioral studies into account and made the decision to utilize gamification to invest in EE behaviors.

Our initial team of designers, researchers and a product manager addressed the core pain points and came up with a phased approach to this feature’s release, starting with a Beta Test in order to collect in-the-field data crucial to our next steps. Our ambitions were to create a strong foundation that embraced scalable functionality and user innovation.

For an extra hand, we proposed this as a research project to the University of Michigan’s Science in Information department and ended up getting valuable research consultation from two Masters students, Candyce Jessamy and Hannah Chen.

The Beta Test went live at one utility and was monitored by the Data Science team to ensure proper functionality. With the O-K from them, we rolled out a 10 week test at other utilities, to which users must opt in.

Internal Research

• ~75% of users engage with the existing Challenges feature on any given month

• This feature is engaged with by users only 2-3 times per week whereas other core features see a rate 7X that

• Customer usage has show about 30% increase between week 1 and 20, despite existing EE challenges

• Upset customer reviews validate our hypotheses

Discovery (related industries)

Various utilities have deployed gamification related EE programs, and multiple competitors have invested resources into building up gamification platforms of their own.

Overall, we found that challenge & reward features are new to the Home Energy Mgmt (HEM) industry, but most offer only Demand Response (DR) challenges and cash rewards.

Discovery (unrelated industries)

To further understand what behavior change techniques are widely used in consumer industries outside of HEM, Candyce and Hannah lead the team in a market research analysis.

We looked into apps that successfully use persuasive technology to gamify their product, such as Duolingo, Headspace, Spotify, Starbucks & Forest, and used behavior change techniques and theoretical framework to standardize the languages while evaluating their techniques.

Takeaways from these apps include: hyper-personalization, social influence & comparison, gamified levels, intrinsic & extrinsic rewards, and goal-setting via alerts.

User Interviews

Frequent users, declining users, & non-users of the current Challenges feature were interviewed to understand user pain points and preferred rewards.

Key findings:

• Users do care about energy conservation

• The current approach to levels is unclear

• Tasks need personalization

• Immediate success feedback is necessary upon completion

• An entertaining progress visualization is nice to have

Focus Groups

A small guided workshop was hosted by U of M + Powerley to expose existing user’s mental models. The results helped prioritize opportunity areas to be used in prototyping.

Lightning Talk, How Might We? (HMW) Affinity Mapping, Crazy Eights, design discussion & concept definition exercises all helped influence these further findings:

• Tasks need to be personalized

• Users like seeing a positive impact on the environment

• Customers want both virtual and tangible rewards

• Social influence features would increase engagement

• Real-time progress visualization is essential

• Suggestion to use this as a channel to promote new features

Card Sorting

Open Card Sorting was conducted, allowing testers to decipher categories that make sense to them from the cards provided. By ranking category importance, testers create a multi-level hierarchy according to how they would like to engage with the feature.

Findings to support the information architecture showed:

• Testers anticipate a full-fledged Reward Center or ‘Shop’

• Tester expect a User Profile and the social comparison abilities

• Status and “bragging rights” were clearly prioritized

Feedback Survey

Utilizing the existing Challenges feature, we prompted AEP Ohio customers provide feedback on challenges they would be interested in completing and the level of effort they’re willing to expend. We found that users:

• Aim to accomplish lowering their utility bill and reducing overall energy use

• Would like to participate in a mix of one-step and multi-step challenges

• Are highly motivated by utility bill discounts, status level, & badges

• A portion of folks would like free smart devices (plugs in particular, to better track their usage)

Ideation

The Design Team iterated what felt like a million wireframes, flows and mocks before arriving at the prototype for a user test.

Mid-Fi Invision Prototype

In order to understand if the architecture of information and terminology supported user-centricity, we asked participants to complete tasks within the prototype and answer questions to discover ease of use. The intention was to validate that the proposed features and tasks would improve engagement and retain users. We found that:

• Testers would like to participate challenges & reward system provided by their utility

• The most-wanted reward is a discount on their utility bill

• We need more general reward options outside of smart devices (cash, giftcards, etc.)

• Testers are motivated by positive reinforcement and intrinsic rewards

• Testers don’t care much about seeing a ‘set’ of intrinsic rewards all in one place

• No testers would share their results to social media

• Overall architecture and concepts resonated

• Participants were able to complete tasks from end-to-end

• The point system was well-understood, though difficulty of a challenge needs to be reflected in the point value

• A lot of uncertainty around ‘level’ status. Users liked the idea of ‘leveling up’ but we're unsure of the structure and benefit

• The majority of testers would never ‘save’ challenges they’d like to complete later

Solutions

The Beta Test had technical constraints due to bandwidth, so we launched Phase 1 with limited requirements. Namely, a bank of 20 reconstructed Challenges that are personalized with home energy, customer journey, & engagement data relevant to the specific user. They must also be feature-toggle aware for differing utility rate types and subscription plans.

Results

This is an ongoing project, but responses from the Beta Test feedback loop have shown:

• 70% of users are more likely to complete challenges under a new system

• On a scale from 1-5, users ranked level of effort to complete challenges at an avg of 2.7

• For most, detailed descriptions made it easier to complete the challenge than before

• The majority of users enjoy tracking with the progress bar, though some did not find the bar to be the most helpful UI element. 10% don’t care about tracking progress at all.

• While completing challenges, 30% of users discovered other features in the app that they were previously unaware of, increasing overall app engagement.

• 20% of users had too much difficulty completing challenges to explore further

Next steps

• Evaluate messaging experimentation for Phase 2 - do positive or negative work better? Self comparative vs. social comparative?

• Meet to discuss VOC analytics and determine the next batch of Challenges

• Inclusion of more intrinsic rewards, and hopefully extrinsic rewards if the utilities are able to supply a budget

• Embed the Challenges experience into other parts of the app, as it can increase effectiveness of other features (e.g. budgeting, home profile)

• Get the challenges system into commercial spaces to further contribute to a distributed grid

• Implement a larger social aspect, such as following friends, competitions, and leaderboards

• Implement an entertaining progress visual (a growing forest, a house being built up, etc.)