On-line testing
Questions this method answers: Highly effective in all business-critical applications involving major decisions relating to creation or re-design of large or complex web sites and web applications. Can access users at home or place of business, 24/7, in fixed or longitudinal study designs. This methodology produces insights on many dimensions that are statistically reliable; can be employed in highly structured or open task sequences. Can reach respondents that would not participate in other testing methods, such as highly placed professionals. Exceptionally powerful when combined with focused lab-based study.
Statistical reliability: moderate-high; large sample size (100-10,000 respondents)
Functional Scope: Many tasks or major interaction sequences possible
Fees: Low cost per subject
Lead time: 3-8 weeks for top line report
Geographic reach: Complete world reach (12 languages)
Deliverable: Comprehensive report covering many dimensions of user interaction, including feature/function optimization, interactive brand attributes ratings, objective and subjective data and skill acquisition profiling. Can be tightly integrated with market research. Sophisticated statistical methods and analysis common.
Major advantage: Fast, reliable, scaleable; international in scope; many user profiles possible; extensive statistical analysis possible; blends well with market research methods; possible in 12 languages.
Disadvantages: No video of user interacting with system. Sometimes low participation rate requires large email list to achieve larger sample sizes. Not totally secure.
Quality of service issues: Some on-line data gathering tools are highly unreliable and data is often inaccurate. Study design and programming is complex and time-consuming if meaningful results are to be obtained. Management of entire study process is complex and requires careful and constant monitoring.
