A 0.3% lift might feel exciting—but if it isn’t statistically significant, it’s just wishful thinking dressed up as data
In fundraising, testing is often described as a best practice. In reality, it is a discipline; one that requires planning, patience and precision.
Too often, organizations test reactively. A new format appears. A vendor suggests a tweak. A board member asks, “Have we tried this?” Testing becomes something we add on rather than something we design strategically.
Smart testing looks very different. It begins with a plan.
Start with a draft testing plan for the year
Effective testing does not happen appeal by appeal. It is mapped out at the beginning of the year.
A draft testing plan outlines what will be tested, in which campaigns, at what scale, with what budget allocation, and most importantly, why.
In the Canadian context, where donor files are often smaller than those of U.S. counterparts, planning becomes even more important. There is less room for casual experimentation. Each test must be intentional.
Every test needs a clear goal
Define success in advance of launching any test.
- What specific metric are you trying to improve?
- By how much would this need to outperform the control to justify rollout?
- What decision will this data inform?
Clear goals prevent post-test rationalization—the temptation to declare something promising simply because it feels innovative.
Test what moves the program forward
Smart testing focuses on elements most likely to drive meaningful performance change, including: offer, ask amounts, segmentation, channel mix and format strategy when justified.
Testing minor cosmetic elements rarely moves net revenue in a significant way. In a resource-constrained environment, energy is better spent on variables with real leverage.
Handle quantities and statistical significance with care
Sample sizes must be large enough to produce statistically meaningful results. Data selects must ensure comparable donor segments. Control groups must remain protected.
In Canada, smaller files can make statistical confidence more challenging. Sometimes tests need to run across multiple campaigns before conclusions can be drawn.
A 0.3% lift might feel exciting — but if it isn’t statistically significant, it’s just wishful thinking dressed up as data.
Apportion costs correctly
If a new format increases response but significantly increases production or acquisition costs, net income may decline. On the other hand, a slightly lower response rate could still produce stronger net revenue if costs are lower.
Testing should never be evaluated solely on gross revenue. Cost allocation ensures apples-to-apples comparisons, particularly when reporting to finance teams and boards.
Protect income while testing robustly
A best practice is to test new approaches on a limited percentage of the file while protecting the majority with a proven control. This allows innovation without jeopardizing essential net income.
Testing is not about dramatic swings. It is about steady, incremental gains that compound over time.
Analyze and interpret results carefully
Careful analysis requires asking whether the test met its primary goal, how it affected secondary metrics, and whether there were unintended consequences. Strong organizations interpret results in context and resist declaring winners too quickly.
Organizations that outperform over time are rarely those making bold, risky leaps. They’re the ones running thoughtful, structured tests year after year, steadily improving performance while protecting core revenue.
Smart testing is not flashy. It is strategic. And, in today’s environment, it may be one of the most important investments a fundraising team can make.
Karri LaJeunesse is a passionate fundraiser with 16 years of experience helping organizations create lasting impact. As a Senior Consultant at Harvey McKinnon Associates, she leads fundraising programs for Oxfam Canada, the Greater Vancouver Food Bank, and the Lions Foundation of Canada Dog Guides. Committed to continuous learning and staying ahead of fundraising trends, Karri thrives on working with mission-driven clients who are making the world a better place.
