I at the moment lead a small information group at a small tech firm. With all the pieces small, we’ve a whole lot of autonomy over what, when, and the way we run experiments. On this collection, I’m opening the vault from our years of experimenting, every story highlighting a key idea associated to experimentation.
And right here we’ll share a shocking outcome from an early take a look at on our referral-bonus program and use it to debate the way you may slender your choice set for experiments (a minimum of after they contain people).
Background: It’s COVID and we have to rent a zillion nurses
IntelyCare helps healthcare amenities match with nursing expertise. We’re a glorified nurse-recruiting machine and so we’re all the time seeking to recruit extra successfully. Nurses come to us from many sources, however those that come through referrals earn greater critiques and stick with us longer.
The yr was 2020. IntelyCare was a child firm (nonetheless is by most requirements). Our app was new and most options have been nonetheless primitive. Some examples…
- We had a means for IntelyPros to share a referral hyperlink with associates however had no monetary incentives to take action.
- Our software course of was a serious slog. We required a small mountain of paperwork for assessment along with a cellphone interview and references. Solely a small subset of candidates made it all through to working.
Throughout a recruiting brainstorm, we latched onto the concept of referrals and agreed that including monetary incentives can be simple to check. One thing like, “Get $100 when your pal begins working.” Zero creativity there, however an concept doesn’t need to be novel to be good.
Realizing that many individuals may refer repeatedly in the event that they earned a bonus, and figuring out that our software course of was nothing wanting a gauntlet, we additionally questioned if it may be higher as a substitute to provide clinicians a small prize when their associates begin an software.
A small prize for one thing simple vs a giant prize for one thing tough? I imply, it relies on many issues. There’s just one strategy to know which is greatest, and that’s to attempt them out.
The referral take a look at
We randomly assigned clinicians to one in every of two experiences:
- The clinician earns an additional $1/hour on their subsequent shift when their referral begins a job software. (Tremendous simple. Beginning an software takes 1–2 minutes.)
- The clinician earns $100 when their referral completes their first shift. (Tremendous laborious. Some nurses race by it, however most candidates take a number of weeks and even months in the event that they end in any respect).
We held out an equal third of clinicians as a management and let clinicians know the principles through a collection of emails. There’s all the time a threat of spillovers in a take a look at like this, however the considered one group stealing all of the referrals from the opposite group appeared like a stretch, so we felt good about randomizing throughout people.
Decidedly non-social: Many individuals hear these two choices and ask, “Did you consider making an attempt prosocial incentives?” (Instance: I refer you, you do one thing, we each get a prize). Research present they’re typically higher than particular person incentives they usually’re fairly frequent (instacart, airbnb, robinhood,…). We thought-about these, however our finance group turned very unhappy on the concept of us sending $1 every to lots of of people that could not ever turn out to be workers.
I suppose Quickbooks doesn’t like that? Sooner or later, you simply settle for that it’s greatest to not mess with the finance group.
Because the $1/hr reward couldn’t be prosocial with out changing into a serious headache, we restricted payouts in each packages to the referring particular person solely. This provides us two referral packages the place the important thing variations are timing and the payout quantity.
Seems timing and presentation of incentives matter. Loads. Social incentives additionally matter. Loads. Should you’re making an attempt to growth-hack your referral program, you’d be sensible to think about each of those dimensions earlier than rising the payout.
Nerdy apart: Fascinated with issues to check
Product information science, with minimal exception, is interested by how people work together with issues. Usually it’s a web site or an app, nevertheless it could possibly be a bodily object like a pair of headphones, a thermostat, or an indication on the freeway.
The science comes from altering the product and watching how people change their conduct consequently. And you’ll’t do higher than watching your clients work together together with your product within the wild to study whether or not a change was useful or not.
However you’ll be able to’t take a look at all the pieces. There are infinite issues to check and any group tasked with experimenting should minimize issues all the way down to a finite set of concepts. The place do you begin?
- Begin with the product itself. Ask people who find themselves accustomed to it how they prefer it, what they want was totally different, Sean Ellis, NPS, the Mother Take a look at, and many others. That is the frequent place to begin for product groups and nearly everybody else.
- Begin with human nature. For a lot of many years Behavioral Scientists have documented explicit patterns in human conduct. These scientists go by totally different names (behavioral economists, behavioral psychologists, and many others.).
In my humble opinion, the 2nd of those beginning factors is severely underrated. Behavioral science has documented dozens of conduct patterns that may inform how your product may change most successfully.
Just a few honorable mentions…
- Loss aversion: folks hate shedding greater than they like successful
- Peak-Finish: folks bear in mind issues extra favorably after they finish on a optimistic notice
- Social vs Market Norms: all the pieces modifications when folks pay for items and companies as a substitute of asking for favors
- Framing: folks make selections primarily based on how info is introduced
- Left-digit Bias: perceptions of a value are disproportionately influenced by the main digit ($0.99 = 🔥, $1.01 = 🥱)
- Current Bias: folks hate ready
These psychological shortcuts aren’t a silver bullet for product and Advertising. We’ve examined many of those in numerous settings to no impact, however some have labored. Possibly you’ll discover $160M underneath your sofa cushions.
Zooming in on current bias
Many people display a conduct sample the place they like small, instant rewards over bigger rewards sooner or later. Social scientists name this current bias and measure it with powerful questions like…
Individuals who select instant, smaller rewards have some measure of current bias.
Current bias can differ throughout folks and circumstances. You might be a affected person individual in most conditions. However once you’re hungry or drained… not a lot. That is one motive why A/B exams are so helpful in comparison with, say, surveys or interviews.
For our experiment, the query shouldn’t be so totally different from the lab questions:
- Would you moderately have ~$8 someday this week or $100 someday within the coming weeks and possibly by no means?
Experiment outcomes
Many people (current firm included) thought the $100 provide would do greatest. It sounds larger. It seems higher in an e-mail. It’s simple to clarify.
I additionally thought we’d get extra purposes from referrals with the $1/hr program, however didn’t suppose that may carry by to extra referrals. I figured folks would take the cash and run.
To my shock the $1/hr program delivered extra purposes through referrals and extra working clinicians, all at a a lot decrease price.
The $100 provide led to a 65% enhance in referrals in comparison with our management group. That is enormous!. Many individuals like this sort of program as a result of we’re solely giving rewards when a referral is profitable. It seems like the cash is effectively spent.
The $1/hr provide, nonetheless, is even higher. Referrals elevated by 81% in comparison with our management group. And although the rewards have been smaller and paid out earlier, referrals from these IntelyPros nonetheless went on to begin working for us at about the identical price because the $100 group.
Sure, many IntelyPros earned rewards for referrals that utilized after which didn’t cross the end line, however the math nonetheless labored out. Even with an imperfect conversion price from software to working, the whole price per new working IntelyPro was lower than half of the $100 group’s price.
ABT: At all times be testing
The function that referrals play for a enterprise can change for a lot of causes. As an organization turns into larger and extra acquainted, referrals turn out to be much less essential. You don’t want folks to unfold the phrase as a result of the phrase is already out.
We ran this take a look at in 2020 throughout a world pandemic. We ran a smaller-scale take a look at a yr later and noticed related outcomes. Would we see the identical outcomes now, in 2025? Arduous to say. Issues are a lot totally different within the healthcare staffing area these days.
That is but one more reason why testing is so essential. One thing like referral bonuses might work wonders for firm Some time being a dud for firm B. The upside of product experiments is that the world is your laboratory. The draw back is that it’s laborious to know what generalizes to different merchandise.
Key takeaways for many who made it this far
- Whenever you’re working experiments to enhance your product and gather information from the people utilizing your product, behavioral science is a fruitful place to search for take a look at concepts.
- Generally the concepts that appear much less nice prove to have the very best influence!
- Timing and presentation matter for incentives. Earlier than you enhance the scale of some payout, maybe put some thought into the effort and time required to earn that payout.
(This put up was refurbished from my very own 2022 put up on behavioraleconomics.com)