Lessons Learned from a year's worth of automated browser testing

My past couple of projects have involved a lot of browser testing.  One of my teammates, Matthieu Tanguay-Carel, inspired by Khan Academy, put together this excellent retrospective video.


Browser testing - looking back from Matt TC on Vimeo.

Testing Strategy

I recently had a chance to sit down with Kristan Vingrys.  He gave me some things to think about in terms of testing strategy.  Here are the things I found the most useful.

Scope
Think big about Quality.  There are many dimensions to quality and each of these may be tested in its own way.

• Non-functional requirements: Does the system scale?, How much traffic can it handle?
• User Experience: Is the search system indexed properly? Are flows intuitive?
• Functional: Do the components play well together?  Does the system have defects?
• Deployment: Do all nodes deploy and load balance reliably?  Is pushing to production a terrifying and drawn-out process?
• Code: Is the code base supple? Self-documenting?  Can I make a change, confident that a unit test will break if I cause a regression?
• Integration: How do we know if an API contract has been violated?

Test Prioritization
He told us to first identify those requirements that, if broken in production, would result in a 3am phone call.  Test these first.

Cost-Benefit Analysis
Automated tests can be expensive to implement.  An effort should be made to only automate tests that make economic sense. 

Automated tests will quickly execute repetitive functional tests much faster and more reliably than a human tester.

• Create an account (Functionality: Fill out and submit form.  Verifying that the submitted data has been handled properly)
• Make a purchase (Functionality: Choose a product, add to cart, fill out credit card information, confirm purchase)

For the time and money involved, human testers are more effective at identifying layout issues, spelling errors, and to some extent, browser compatibility issues.  In most cases, these are non-critical issues.

• Text doesn't overrun a border
• The name of a section is spelled correctly
• The wording of legal text
• Styling of page elements
• The look is the same in Firefox and IE6