A/B Testing for Digital Signage
Most digital signage content is created based on assumptions and opinions. A/B testing replaces guesswork with data, allowing you to scientifically determine what content drives the best results. This guide covers methodology, metrics, and practical implementation for signage optimization.
Why A/B Test Digital Signage?
The Impact of Optimization
| Metric | Unoptimized | After A/B Testing |
|---|---|---|
| Viewer attention rate | 25% | 40-60% |
| Call-to-action response | 2% | 5-8% |
| Promotional lift | 5% | 15-25% |
| Message recall | 15% | 35-50% |
Common Assumptions That Are Wrong
| Assumption | Reality (from testing) |
|---|---|
| "Bigger text is always better" | Depends on viewing context |
| "Video outperforms static" | Not always - complexity matters |
| "Red calls attention" | Can signal danger, reduce action |
| "More info is better" | Often decreases comprehension |
| "Our brand colors work best" | Sometimes neutral performs better |
A/B Testing Fundamentals
What is A/B Testing?
┌─────────────────────────────────────────────────────────────────────────┐
│ A/B TEST STRUCTURE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ CONTROL (A) VARIANT (B) │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ │ │ │ │
│ │ Current Design │ │ New Design │ │
│ │ │ │ (One Change) │ │
│ │ │ │ │ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ 50% of displays │ │ 50% of displays │ │
│ │ or time │ │ or time │ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │ │ │
│ └──────────────┬───────────────┘ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ COMPARE RESULTS │ │
│ │ Statistical │ │
│ │ Significance │ │
│ └─────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ IMPLEMENT │ │
│ │ WINNER │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Key Testing Principles
- Test one variable at a time - Otherwise you can't attribute results
- Adequate sample size - Enough exposure for statistical validity
- Run simultaneously - Eliminates time-based variables
- Define metrics beforehand - Know what success looks like
- Achieve statistical significance - Don't stop early
What to Test
High-Impact Test Variables
| Variable | Impact Potential | Test Difficulty |
|---|---|---|
| Headlines/Copy | Very High | Easy |
| Call-to-Action | Very High | Easy |
| Hero Image | High | Easy |
| Price Display | High | Easy |
| Layout | High | Medium |
| Color Scheme | Medium-High | Easy |
| Animation | Medium | Medium |
| Content Duration | Medium | Easy |
| Font Size | Medium | Easy |
| Background | Low-Medium | Easy |
Headline/Copy Tests
Test variations in:
| Element | Variant A | Variant B |
|---|---|---|
| Benefit focus | "Save 50%" | "Half Price" |
| Urgency | "Today Only" | "Limited Time" |
| Question vs. Statement | "Hungry?" | "Satisfy Your Craving" |
| Length | "Get 50% Off All Items" | "50% Off" |
| Specificity | "Save Money" | "Save $10 Today" |
| Tone | "Amazing Deal" | "Smart Choice" |
Call-to-Action Tests
| Element | Variant A | Variant B |
|---|---|---|
| Action verb | "Buy Now" | "Shop Now" |
| Urgency | "Order Today" | "Don't Miss Out" |
| Benefit | "Get Yours" | "Start Saving" |
| Question | "Ready to Save?" | "Save Now" |
| Position | Top of screen | Bottom of screen |
| Size | Standard | 20% larger |
| Color | Brand color | Contrasting color |
Image Tests
| Element | Test Options |
|---|---|
| Subject | Product alone vs. product in use |
| People | With people vs. without |
| Angle | Close-up vs. wide shot |
| Mood | Bright/energetic vs. calm/sophisticated |
| Quantity | Single item vs. collection |
| Background | Plain vs. lifestyle context |
Layout Tests
┌─────────────────────────────────────────────────────────────────┐
│ LAYOUT TEST EXAMPLE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ VARIANT A: Image Left VARIANT B: Image Right │
│ ┌─────────┬─────────────┐ ┌─────────────┬─────────┐ │
│ │ │ Headline │ │ Headline │ │ │
│ │ IMAGE │ Body text │ │ Body text │ IMAGE │ │
│ │ │ CTA Button │ │ CTA Button │ │ │
│ └─────────┴─────────────┘ └─────────────┴─────────┘ │
│ │
│ VARIANT C: Image Top VARIANT D: Full Bleed │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ IMAGE │ │█████████████████████│ │
│ ├─────────────────────┤ │██ Headline ██████████│ │
│ │ Headline │ │██ Body text █████████│ │
│ │ Body text CTA │ │██ CTA Button ████████│ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
Metrics for Digital Signage A/B Tests
Primary Metrics
| Metric | How to Measure | Best For |
|---|---|---|
| Attention Rate | Camera/sensor: viewers ÷ passers | Engagement optimization |
| Dwell Time | Camera/sensor: seconds looking | Content interest |
| QR Code Scans | Scan count per impression | Direct response |
| Promotional Lift | POS: promo sales vs. baseline | Sales content |
| Conversion Rate | Actions ÷ impressions | Call-to-action |
Secondary Metrics
| Metric | How to Measure | Use Case |
|---|---|---|
| Traffic Flow | Sensors: direction toward display | Wayfinding |
| Interaction Rate | Touch/gesture count | Interactive content |
| Session Duration | Touch: time in session | Kiosk engagement |
| Survey Response | Post-exposure surveys | Message recall |
| Social Mentions | Hashtag/mention tracking | Brand campaigns |
Calculating Key Metrics
Attention Rate:
Attention Rate = (Viewers Looking at Screen ÷ Total Passers) × 100
Example: 250 viewers ÷ 1000 passers = 25% attention rate
Promotional Lift:
Lift = ((Test Period Sales - Baseline Sales) ÷ Baseline Sales) × 100
Example: ($15,000 - $10,000) ÷ $10,000 = 50% lift
Statistical Significance:
For 95% confidence (p < 0.05):
- Need sufficient sample size
- Difference must exceed margin of error
- Use chi-square or t-test
Test Design & Methodology
Sample Size Calculator
| Baseline Rate | Minimum Lift to Detect | Sample Size Needed (per variant) |
|---|---|---|
| 5% | 20% relative (5%→6%) | 15,000 |
| 5% | 50% relative (5%→7.5%) | 2,500 |
| 10% | 20% relative (10%→12%) | 4,000 |
| 10% | 50% relative (10%→15%) | 700 |
| 25% | 20% relative (25%→30%) | 1,100 |
| 25% | 50% relative (25%→37.5%) | 200 |
Rule of thumb: Aim for at least 1,000 observations per variant minimum.
Test Duration Guidelines
| Factor | Consideration |
|---|---|
| Traffic volume | Low traffic = longer test |
| Day-of-week effects | Run full weeks to capture patterns |
| Seasonality | Avoid holidays unless testing for them |
| Promotional cycles | Test outside major promotions |
| Minimum duration | At least 1-2 weeks |
| Maximum duration | 4-6 weeks before fatigue |
Splitting Strategy
Option 1: Time-Based Split
Mon-Wed: Variant A
Thu-Sat: Variant B
(Rotate next week)
- ✅ Simple to implement
- ❌ Day-of-week bias possible
Option 2: Location Split
Stores 1, 3, 5: Variant A
Stores 2, 4, 6: Variant B
- ✅ Simultaneous testing
- ❌ Location differences may confound
Option 3: Display Split
Screen 1: Variant A
Screen 2: Variant B
(Same location)
- ✅ Controls for location
- ❌ Needs multiple screens
Option 4: Random Rotation
Each play: Random A or B
(50/50 probability)
- ✅ Best for statistical validity
- ❌ Requires CMS support
Running a Test: Step by Step
Phase 1: Planning (Week 1)
┌─────────────────────────────────────────────────────────────────┐
│ TEST PLANNING CHECKLIST │
├─────────────────────────────────────────────────────────────────┤
│ │
│ □ Define hypothesis │
│ "Changing X will improve Y by Z%" │
│ │
│ □ Select single variable to test │
│ │
│ □ Define primary success metric │
│ │
│ □ Calculate required sample size │
│ │
│ □ Determine test duration │
│ │
│ □ Create both variants (A and B) │
│ │
│ □ Set up measurement/tracking │
│ │
│ □ Document current baseline performance │
│ │
│ □ Get stakeholder buy-in │
│ │
└─────────────────────────────────────────────────────────────────┘
Phase 2: Execution (Weeks 2-3)
- Launch simultaneously: Both variants start at same time
- Monitor for errors: Check displays, tracking, data collection
- Don't peek: Avoid making decisions on early data
- Document issues: Record any anomalies
- Maintain consistency: Don't change other variables
Phase 3: Analysis (Week 4)
Analysis Template:
TEST: [Name]
HYPOTHESIS: [Statement]
DATES: [Start] - [End]
DISPLAYS: [List]
RESULTS:
VARIANT A VARIANT B
Impressions: 10,000 10,000
Viewers: 2,500 3,200
Attention Rate: 25% 32%
Difference: +7 percentage points (+28% relative)
STATISTICAL TEST:
Chi-square value: [X]
p-value: [Y]
Confidence: [Z]%
WINNER: Variant B
RECOMMENDATION: Implement Variant B across all locations
NEXT TEST: [Idea]
Phase 4: Implementation
- Roll out winner to all displays
- Document learnings for future reference
- Monitor post-implementation for consistency
- Plan next test based on learnings
Test Examples & Results
Example 1: Headline Test
Hypothesis: Urgency-focused headline will increase attention rate
| Variant A (Control) | Variant B (Urgency) | |
|---|---|---|
| Headline | "Summer Collection" | "Last Days of Summer Sale" |
| Impressions | 5,000 | 5,000 |
| Attention Rate | 22% | 31% |
| Result | +41% improvement |
Example 2: Image Test
Hypothesis: People in images increase engagement
| Variant A (Product Only) | Variant B (With People) | |
|---|---|---|
| Image | Shoes on white | Model wearing shoes |
| Dwell Time | 2.1 seconds | 3.8 seconds |
| Result | +81% improvement |
Example 3: CTA Color Test
Hypothesis: Contrasting CTA button improves scans
| Variant A (Brand Blue) | Variant B (Orange) | |
|---|---|---|
| CTA Color | #0066CC | #FF6600 |
| QR Scans | 45 | 78 |
| Scan Rate | 0.9% | 1.6% |
| Result | +78% improvement |
Example 4: Animation Test
Hypothesis: Subtle animation increases attention
| Variant A (Static) | Variant B (Animated) | |
|---|---|---|
| Treatment | Static image | Gentle zoom effect |
| Attention Rate | 28% | 34% |
| Dwell Time | 2.5 sec | 2.2 sec |
| Result | More attention, less dwell |
Learning: Animation grabbed attention but didn't hold it. Static may be better for conveying information.
Common Mistakes to Avoid
| Mistake | Problem | Solution |
|---|---|---|
| Testing too many variables | Can't attribute results | One change at a time |
| Stopping early | False positives | Wait for significance |
| Ignoring external factors | Confounded results | Control for variables |
| No baseline | Can't measure improvement | Document current state |
| Small sample size | Unreliable results | Calculate needs upfront |
| Confirmation bias | See what you want | Pre-define metrics |
| Not documenting | Lost learnings | Keep detailed records |
| Testing trivial changes | Wasted effort | Focus on high-impact |
Building a Testing Culture
Testing Roadmap
┌─────────────────────────────────────────────────────────────────┐
│ ANNUAL TESTING ROADMAP │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Q1: Foundation Tests │
│ ├── Headline formulation tests │
│ ├── Primary CTA optimization │
│ └── Core layout testing │
│ │
│ Q2: Content Type Tests │
│ ├── Static vs. video │
│ ├── Animation effectiveness │
│ └── Information density │
│ │
│ Q3: Optimization Tests │
│ ├── Winning element combinations │
│ ├── Timing and dayparting │
│ └── Seasonal content │
│ │
│ Q4: Advanced Tests │
│ ├── Personalization approaches │
│ ├── Interactive elements │
│ └── Multi-screen coordination │
│ │
│ Target: 12-24 tests per year │
│ Goal: 10-15% annual performance improvement │
│ │
└─────────────────────────────────────────────────────────────────┘
Learning Library
Document all tests:
- Test name and date
- Hypothesis
- Variables tested
- Results and winner
- Statistical confidence
- Key learnings
- Recommendations
Sharing Results
| Audience | Focus |
|---|---|
| Executives | ROI impact, strategic learnings |
| Marketing | Creative insights, best practices |
| Operations | Implementation requirements |
| Design team | Visual guidelines that work |
Frequently Asked Questions
Try it on your own screens, free
DigitalSignage.com, which publishes this guide, runs a permanent free plan: the first 3 screens are free forever (no credit card, no ads, no time limit), then from $3 per screen per month with volume pricing via a public calculator. The free SignPlayer runs on Windows, Mac, Android and Android TV, Chrome OS, Raspberry Pi, iPad or any modern web browser. Start free · 2026 pricing
Next Steps
- Audience Analytics - Measurement technology
- Content Best Practices - Design guidelines
- Data-Driven Content - Dynamic optimization
- ROI Calculator - Measure business impact
This guide is maintained by MediaSignage, pioneers of digital signage technology since 2006.