A/B Testing for Digital Signage
Most digital signage content is created based on assumptions and opinions. A/B testing replaces guesswork with data, allowing you to scientifically determine what content drives the best results. This guide covers methodology, metrics, and practical implementation for signage optimization.
Why A/B Test Digital Signage?
The Impact of Optimization
| Metric | Unoptimized | After A/B Testing |
|---|---|---|
| Viewer attention rate | 25% | 40-60% |
| Call-to-action response | 2% | 5-8% |
| Promotional lift | 5% | 15-25% |
| Message recall | 15% | 35-50% |
Common Assumptions That Are Wrong
| Assumption | Reality (from testing) |
|---|---|
| "Bigger text is always better" | Depends on viewing context |
| "Video outperforms static" | Not always - complexity matters |
| "Red calls attention" | Can signal danger, reduce action |
| "More info is better" | Often decreases comprehension |
| "Our brand colors work best" | Sometimes neutral performs better |
A/B Testing Fundamentals
What is A/B Testing?
┌─────────────────────────────────────────────────────────────────────────┐
│ A/B TEST STRUCTURE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ CONTROL (A) VARIANT (B) │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ │ │ │ │
│ │ Current Design │ │ New Design │ │
│ │ │ │ (One Change) │ │
│ │ │ │ │ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ 50% of displays │ │ 50% of displays │ │
│ │ or time │ │ or time │ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │ │ │
│ └──────────────┬───────────────┘ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ COMPARE RESULTS │ │
│ │ Statistical │ │
│ │ Significance │ │
│ └─────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ IMPLEMENT │ │
│ │ WINNER │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Key Testing Principles
- Test one variable at a time - Otherwise you can't attribute results
- Adequate sample size - Enough exposure for statistical validity
- Run simultaneously - Eliminates time-based variables
- Define metrics beforehand - Know what success looks like
- Achieve statistical significance - Don't stop early
What to Test
High-Impact Test Variables
| Variable | Impact Potential | Test Difficulty |
|---|---|---|
| Headlines/Copy | Very High | Easy |
| Call-to-Action | Very High | Easy |
| Hero Image | High | Easy |
| Price Display | High | Easy |
| Layout | High | Medium |
| Color Scheme | Medium-High | Easy |
| Animation | Medium | Medium |
| Content Duration | Medium | Easy |
| Font Size | Medium | Easy |
| Background | Low-Medium | Easy |
Headline/Copy Tests
Test variations in:
| Element | Variant A | Variant B |
|---|---|---|
| Benefit focus | "Save 50%" | "Half Price" |
| Urgency | "Today Only" | "Limited Time" |
| Question vs. Statement | "Hungry?" | "Satisfy Your Craving" |
| Length | "Get 50% Off All Items" | "50% Off" |
| Specificity | "Save Money" | "Save $10 Today" |
| Tone | "Amazing Deal" | "Smart Choice" |
Call-to-Action Tests
| Element | Variant A | Variant B |
|---|---|---|
| Action verb | "Buy Now" | "Shop Now" |
| Urgency | "Order Today" | "Don't Miss Out" |
| Benefit | "Get Yours" | "Start Saving" |
| Question | "Ready to Save?" | "Save Now" |
| Position | Top of screen | Bottom of screen |
| Size | Standard | 20% larger |
| Color | Brand color | Contrasting color |
Image Tests
| Element | Test Options |
|---|---|
| Subject | Product alone vs. product in use |
| People | With people vs. without |
| Angle | Close-up vs. wide shot |
| Mood | Bright/energetic vs. calm/sophisticated |
| Quantity | Single item vs. collection |
| Background | Plain vs. lifestyle context |
Layout Tests
┌─────────────────────────────────────────────────────────────────┐
│ LAYOUT TEST EXAMPLE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ VARIANT A: Image Left VARIANT B: Image Right │
│ ┌─────────┬─────────────┐ ┌─────────────┬─────────┐ │
│ │ │ Headline │ │ Headline │ │ │
│ │ IMAGE │ Body text │ │ Body text │ IMAGE │ │
│ │ │ CTA Button │ │ CTA Button │ │ │
│ └─────────┴─────────────┘ └─────────────┴─────────┘ │
│ │
│ VARIANT C: Image Top VARIANT D: Full Bleed │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ IMAGE │ │█████████████████████│ │
│ ├─────────────────────┤ │██ Headline ██████████│ │
│ │ Headline │ │██ Body text █████████│ │
│ │ Body text CTA │ │██ CTA Button ████████│ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
Metrics for Digital Signage A/B Tests
Primary Metrics
| Metric | How to Measure | Best For |
|---|---|---|
| Attention Rate | Camera/sensor: viewers ÷ passers | Engagement optimization |
| Dwell Time | Camera/sensor: seconds looking | Content interest |
| QR Code Scans | Scan count per impression | Direct response |
| Promotional Lift | POS: promo sales vs. baseline | Sales content |
| Conversion Rate | Actions ÷ impressions | Call-to-action |
Secondary Metrics
| Metric | How to Measure | Use Case |
|---|---|---|
| Traffic Flow | Sensors: direction toward display | Wayfinding |
| Interaction Rate | Touch/gesture count | Interactive content |
| Session Duration | Touch: time in session | Kiosk engagement |
| Survey Response | Post-exposure surveys | Message recall |
| Social Mentions | Hashtag/mention tracking | Brand campaigns |
Calculating Key Metrics
Attention Rate:
Attention Rate = (Viewers Looking at Screen ÷ Total Passers) × 100
Example: 250 viewers ÷ 1000 passers = 25% attention rate
Promotional Lift:
Lift = ((Test Period Sales - Baseline Sales) ÷ Baseline Sales) × 100
Example: ($15,000 - $10,000) ÷ $10,000 = 50% lift
Statistical Significance:
For 95% confidence (p < 0.05):
- Need sufficient sample size
- Difference must exceed margin of error
- Use chi-square or t-test
Test Design & Methodology
Sample Size Calculator
| Baseline Rate | Minimum Lift to Detect | Sample Size Needed (per variant) |
|---|---|---|
| 5% | 20% relative (5%→6%) | 15,000 |
| 5% | 50% relative (5%→7.5%) | 2,500 |
| 10% | 20% relative (10%→12%) | 4,000 |
| 10% | 50% relative (10%→15%) | 700 |
| 25% | 20% relative (25%→30%) | 1,100 |
| 25% | 50% relative (25%→37.5%) | 200 |
Rule of thumb: Aim for at least 1,000 observations per variant minimum.
Test Duration Guidelines
| Factor | Consideration |
|---|---|
| Traffic volume | Low traffic = longer test |
| Day-of-week effects | Run full weeks to capture patterns |
| Seasonality | Avoid holidays unless testing for them |
| Promotional cycles | Test outside major promotions |
| Minimum duration | At least 1-2 weeks |
| Maximum duration | 4-6 weeks before fatigue |
Splitting Strategy
Option 1: Time-Based Split
Mon-Wed: Variant A
Thu-Sat: Variant B
(Rotate next week)
- ✅ Simple to implement
- ❌ Day-of-week bias possible
Option 2: Location Split
Stores 1, 3, 5: Variant A
Stores 2, 4, 6: Variant B
- ✅ Simultaneous testing
- ❌ Location differences may confound
Option 3: Display Split
Screen 1: Variant A
Screen 2: Variant B
(Same location)
- ✅ Controls for location
- ❌ Needs multiple screens
Option 4: Random Rotation
Each play: Random A or B
(50/50 probability)
- ✅ Best for statistical validity
- ❌ Requires CMS support
Running a Test: Step by Step
Phase 1: Planning (Week 1)
┌─────────────────────────────────────────────────────────────────┐
│ TEST PLANNING CHECKLIST │
├─────────────────────────────────────────────────────────────────┤
│ │
│ □ Define hypothesis │
│ "Changing X will improve Y by Z%" │
│ │
│ □ Select single variable to test │
│ │
│ □ Define primary success metric │
│ │
│ □ Calculate required sample size │
│ │
│ □ Determine test duration │
│ │
│ □ Create both variants (A and B) │
│ │
│ □ Set up measurement/tracking │
│ │
│ □ Document current baseline performance │
│ │
│ □ Get stakeholder buy-in │
│ │
└─────────────────────────────────────────────────────────────────┘
Phase 2: Execution (Weeks 2-3)
- Launch simultaneously: Both variants start at same time
- Monitor for errors: Check displays, tracking, data collection
- Don't peek: Avoid making decisions on early data
- Document issues: Record any anomalies
- Maintain consistency: Don't change other variables
Phase 3: Analysis (Week 4)
Analysis Template:
TEST: [Name]
HYPOTHESIS: [Statement]
DATES: [Start] - [End]
DISPLAYS: [List]
RESULTS:
VARIANT A VARIANT B
Impressions: 10,000 10,000
Viewers: 2,500 3,200
Attention Rate: 25% 32%
Difference: +7 percentage points (+28% relative)
STATISTICAL TEST:
Chi-square value: [X]
p-value: [Y]
Confidence: [Z]%
WINNER: Variant B
RECOMMENDATION: Implement Variant B across all locations
NEXT TEST: [Idea]
Phase 4: Implementation
- Roll out winner to all displays
- Document learnings for future reference
- Monitor post-implementation for consistency
- Plan next test based on learnings
Test Examples & Results
Example 1: Headline Test
Hypothesis: Urgency-focused headline will increase attention rate
| Variant A (Control) | Variant B (Urgency) | |
|---|---|---|
| Headline | "Summer Collection" | "Last Days of Summer Sale" |
| Impressions | 5,000 | 5,000 |
| Attention Rate | 22% | 31% |
| Result | +41% improvement |
Example 2: Image Test
Hypothesis: People in images increase engagement
| Variant A (Product Only) | Variant B (With People) | |
|---|---|---|
| Image | Shoes on white | Model wearing shoes |
| Dwell Time | 2.1 seconds | 3.8 seconds |
| Result | +81% improvement |
Example 3: CTA Color Test
Hypothesis: Contrasting CTA button improves scans
| Variant A (Brand Blue) | Variant B (Orange) | |
|---|---|---|
| CTA Color | #0066CC | #FF6600 |
| QR Scans | 45 | 78 |
| Scan Rate | 0.9% | 1.6% |
| Result | +78% improvement |
Example 4: Animation Test
Hypothesis: Subtle animation increases attention
| Variant A (Static) | Variant B (Animated) | |
|---|---|---|
| Treatment | Static image | Gentle zoom effect |
| Attention Rate | 28% | 34% |
| Dwell Time | 2.5 sec | 2.2 sec |
| Result | More attention, less dwell |
Learning: Animation grabbed attention but didn't hold it. Static may be better for conveying information.
Common Mistakes to Avoid
| Mistake | Problem | Solution |
|---|---|---|
| Testing too many variables | Can't attribute results | One change at a time |
| Stopping early | False positives | Wait for significance |
| Ignoring external factors | Confounded results | Control for variables |
| No baseline | Can't measure improvement | Document current state |
| Small sample size | Unreliable results | Calculate needs upfront |
| Confirmation bias | See what you want | Pre-define metrics |
| Not documenting | Lost learnings | Keep detailed records |
| Testing trivial changes | Wasted effort | Focus on high-impact |
Building a Testing Culture
Testing Roadmap
┌─────────────────────────────────────────────────────────────────┐
│ ANNUAL TESTING ROADMAP │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Q1: Foundation Tests │
│ ├── Headline formulation tests │
│ ├── Primary CTA optimization │
│ └── Core layout testing │
│ │
│ Q2: Content Type Tests │
│ ├── Static vs. video │
│ ├── Animation effectiveness │
│ └── Information density │
│ │
│ Q3: Optimization Tests │
│ ├── Winning element combinations │
│ ├── Timing and dayparting │
│ └── Seasonal content │
│ │
│ Q4: Advanced Tests │
│ ├── Personalization approaches │
│ ├── Interactive elements │
│ └── Multi-screen coordination │
│ │
│ Target: 12-24 tests per year │
│ Goal: 10-15% annual performance improvement │
│ │
└─────────────────────────────────────────────────────────────────┘
Learning Library
Document all tests:
- Test name and date
- Hypothesis
- Variables tested
- Results and winner
- Statistical confidence
- Key learnings
- Recommendations
Sharing Results
| Audience | Focus |
|---|---|
| Executives | ROI impact, strategic learnings |
| Marketing | Creative insights, best practices |
| Operations | Implementation requirements |
| Design team | Visual guidelines that work |
Frequently Asked Questions
How long should I run a digital signage A/B test?
Minimum 1-2 weeks to account for day-of-week variations, ideally 2-4 weeks for reliable results. The exact duration depends on your traffic volume and the size of difference you're trying to detect. Run until you achieve statistical significance (p < 0.05) with adequate sample size.
Can I test multiple variables at once?
It's best to test one variable at a time so you can clearly attribute any difference to that change. If you must test multiple variables, use multivariate testing methodology with larger sample sizes. However, for most digital signage, sequential A/B tests are more practical and actionable.
What's a good sample size for digital signage A/B tests?
Aim for at least 1,000 observations per variant as a minimum. For detecting smaller differences (10-20% improvement), you may need 5,000-10,000 per variant. Use a sample size calculator based on your baseline conversion rate and the minimum effect size you want to detect.
How do I measure digital signage A/B test results without cameras?
Without audience measurement cameras, use proxy metrics: QR code scans, promotional sales lift, coupon redemptions, website traffic spikes, or survey responses. For restaurants, compare item sales during test periods. The key is having a measurable action connected to your signage content.
Try it on your own screens, free
DigitalSignage.com, which publishes this guide, runs a permanent free plan: the first 3 screens are free forever (no credit card, no ads, no time limit), then from $3 per screen per month with volume pricing via a public calculator. The free SignPlayer runs on Windows, Mac, Android and Android TV, Chrome OS, Raspberry Pi, iPad or any modern web browser. Start free · 2026 pricing
Next Steps
- Audience Analytics - Measurement technology
- Content Best Practices - Design guidelines
- Data-Driven Content - Dynamic optimization
- ROI Calculator - Measure business impact
This guide is maintained by MediaSignage, pioneers of digital signage technology since 2006.