More and more web teams are incorporating usability testing into their work—that’s a good thing, but it’s time to take our game to the next level.

Usability testing is not a magic bullet. The process generates questions without answering them. Solutions are then dispensed, broadly, drawn from your expert’s long experience. How do you separate sage advice from benevolent bullshit?

High school science classes are taught the scientific method: Ask a question, construct a hypothesis, conduct an experiment, analyze results, and draw a conclusion. It’s the backbone of modern research.

Experiments are present, but uncommon, in usability research. Jakob Nielsen says the benefits of A/B testing (a type of experiment) are outweighed by qualitative observation. Given a limited pool of money, Nielsen thinks you’re better off spending on usability testing. He’s right, but he wrote that in 2005. Six years later, the user experience field has grown up.

There’s a disturbing lack of practical usability experiments in research journals. I’ve been digging through them for years without much success. Outside of Jakob Nielsen’s Alertbox column, you won’t find many blogs talking about usability research, either. That’s one of the reasons I focused this site on substance instead of fluff.

Within five years, experiments will be a cornerstone of user experience research. Conventions and best practices will provide a usability baseline.  Observational research will generate basic questions that are answered by logic and experience. For the gray areas and questions outside our blanket of knowledge, we’ll design experiments. And hopefully, the findings will be shared.