Lessons Learned from a year's worth of automated browser testing

My past couple of projects have involved a lot of browser testing.  One of my teammates, Matthieu Tanguay-Carel, inspired by Khan Academy, put together this excellent retrospective video.


Browser testing - looking back from Matt TC on Vimeo.

Testing Strategy

I recently had a chance to sit down with Kristan Vingrys.  He gave me some things to think about in terms of testing strategy.  Here are the things I found the most useful.

Scope
Think big about Quality.  There are many dimensions to quality and each of these may be tested in its own way.

• Non-functional requirements: Does the system scale?, How much traffic can it handle?
• User Experience: Is the search system indexed properly? Are flows intuitive?
• Functional: Do the components play well together?  Does the system have defects?
• Deployment: Do all nodes deploy and load balance reliably?  Is pushing to production a terrifying and drawn-out process?
• Code: Is the code base supple? Self-documenting?  Can I make a change, confident that a unit test will break if I cause a regression?
• Integration: How do we know if an API contract has been violated?

Test Prioritization
He told us to first identify those requirements that, if broken in production, would result in a 3am phone call.  Test these first.

Cost-Benefit Analysis
Automated tests can be expensive to implement.  An effort should be made to only automate tests that make economic sense. 

Automated tests will quickly execute repetitive functional tests much faster and more reliably than a human tester.

• Create an account (Functionality: Fill out and submit form.  Verifying that the submitted data has been handled properly)
• Make a purchase (Functionality: Choose a product, add to cart, fill out credit card information, confirm purchase)

For the time and money involved, human testers are more effective at identifying layout issues, spelling errors, and to some extent, browser compatibility issues.  In most cases, these are non-critical issues.

• Text doesn't overrun a border
• The name of a section is spelled correctly
• The wording of legal text
• Styling of page elements
• The look is the same in Firefox and IE6

Levels of Testing

A lot of people ask me if Functional testing replaces Unit testing and Test Driven Development (TDD).  My answer is no and here is my reasoning:

Unit tests verify that discreet units of logic are sound.

Integration tests ensure that the integrated parties adhere to the same contract.

Functional tests verify that the app as a whole performs as expected, from the perspective of the end user.

Using a water bottle as an example:

Unit test:

  The bottle should not leak when filled with water.

  The bottle should hold 12oz of liquid.

  Bottle & Cap material should be non-toxic.

  The cap should have no sharp edges.

Integration test:

  Both cap and bottle adhere to a 1" diameter, clockwise thread.

  When fastened, the cap and bottle should form a tight seal.

Functional test:

  Given that the bottle is full and the cap is fastened, a drop from 6' should not cause a spill.

  The capped-bottle shouldn't leak when heated or cooled within a normal range of temperatures.

Each works at a different level of abstraction.  While there is some overlap in test coverage, none is replaceable by the others.  Functional tests help to ensure that the whole app hangs together ok, but they don't give you the same visibility as a good, granular unit test.  

For example, the heating-cooling functional test would fail if the bottle leaked.  However, it would take some debugging to get to figure out whether the cap, bottle, or the combination of the two was at the root of the problem.  Good unit tests will alert us to any bottle-specific issues way earlier, eliminating the need to debug the fully deployed system.  

That said, I always try to lock in the desired behavior at the unit level first and lean on functional testing for things such as AJAX that don't unit test very well.

Behavior Driven Development

College was misleading.  There, we were led to believe that the customer (professor) was technical and actually knew in advance exactly what they wanted.  The requirements were chiseled in stone, two months out, and there were test suites waiting to tell us if our work was "correct."  

In reality, the customer is generally an expert at something other than technology.  They generally don't think in terms of edge cases and they certainly don't write tests for us in advance to ensure that we cover them.  Even if they were, business is a fast-paced and evolving thing, what is seen to be a competitive advantage today may be a liability tomorrow.  The only constant is change.

Agile teams understand this and structure their approach to delivery with this in mind.  Rather than attempt to conceive, build, and deliver the perfect application in one fell swoop, they steadily deliver work in small, usable chunks, every few weeks.  The customer is encouraged to think in terms of what functionality will deliver the maximum benefit to the business and prioritize the pipeline of work accordingly.

This tends to reduce feature creep, but it doesn't address the generally poor communication between the project's various stakeholders and the resultantly poor-quality deliverable:

This is where BDD comes in.  It bakes quality into the process by smoothing communication between stakeholders.  Here is how we're practicing it on my current project:

• The Business Analyst works with the Customer to determine what it is that the business is trying to accomplish.

• The BA, QA, and Developer sit down and have a discussion. The QA focuses on flushing out the bounds of the system and edge cases.  The Dev focuses on implementation and brings up potential technical hurdles.  The Business Analyst answers questions, ensuring that the customer's high-level business needs are met.  This is the software equivalent of 'measure twice, cut once.'  Many times, poorly conceived requirements are eliminated before even reaching development  

The outcome of this meeting is a document that will function as a specification, test suite, and living system documentation. Here is a simple Login Story written in Given, When, Then format:

Login Story

Scenario:  Basic Login

Given I am a basic user

When I login

Then I should see the basic homepage

Scenario:  Administrative Login

Given I am an administrative user

When I login

Then I should see the administrative homepage

Scenario:  Invalid Login

Given I am not a valid user

When I login

Then I should see the error page

• The developer(s) now get to work on the implementation of the story.  The preceding conversation will have given them a high-level understanding of what the business wants from the system as well as clearly codified acceptance criteria.  Their high-level perspective prevents them from diving down a tangential rabbit-hole while the non-prescriptive nature of the spec provides for creative latitude in implementation.

• The feature complete, the QA and Developer pair on functional test implementation.  The developer brings technical expertise, while the QA brings an understanding of the system as a whole.  They "fix", to each step in their story, code that will exercise the new system features.

• Next comes what we call the "desk check." The pair calls the BA over and demonstrates the completed feature by running the functional test.  The BA may give an initial sign-off there or they may notice something incorrect  or incomplete.  In the case of the former, the dev will move on to a new story; in the latter, the dev will correct the behavior of the system, update the functional test to "lock-in" the additional requirement, and re-submit for approval.  This process has the benefit that the developer has not yet mentally context-switched to a different piece of work and so they will be able to more quickly rectify the problem than if they worked on the fix weeks later after it gets logged as a bug.

• When, at some later date, the business inevitably decides to change a system requirement, the test will fail, as the system is no longer performing as expected.  The story is then modified to reflect the new behavior.  Using our previous example:

(Original)

Scenario:  Invalid Login

Given I am not a valid user

When I login

Then I should see the error page

(New)

Scenario:  Invalid Login

Given I am not a valid user

When I login

Then I should see an error message on the login page

In this way, we say that the story functions as a living document.  As the behavior of the system evolves over time, the team is forced to evolve the documentation in parallel.

Why Agile?

I am primarily interested in problem solving. Some problems are technical in nature, others are social. I like that the Agile approach to software development embraces human nature and leverages it to the benefit of both the business and it's employees.


An Example:
In the interest of limiting distractions, many companies build cube farms.  They tuck each of their employees away in their own little box and assume that this will make them more productive.  This is a technical solution to a social problem.  If we were horses, blinders might work, but we're not.  We are social monkeys.  We need social interaction and mental stimulation, so we then turn to facebook or the web when we get lonely or bored.  The company then employs website filtering in an attempt to keep the employees on task.  This results in frustration as employees are prevented from doing legitimate research as a result of the filter algorithm's false positive's.  As time progresses, the employee grows to resent their overlord.  They build relatively weak social relationships with their co-workers and they feel no sense of loss when they quit.


Now let's look at the Agile (XP) alternative, Pair-Programming.  Two people, two keyboards, two mice, one computer.  The entire workday is a combination of conversation and problem-solving.  It's anything but boring; "Let's do this", "You're an idiot", "Holy crap, you're right!"  It is difficult to surf the web when you're in the midst of a conversation.  Over the course of a few days worth of shared coffee breaks, lunches, and late night beers, you develop some surprisingly deep friendships.  Over the course of a project, the team becomes a family.  Pair-Programming is a social solution to a set of social problems.  It reinforces positive behaviors without impugning the dignity of the employee.  On top of that, the resulting code has 40% fewer bugs, requires fewer total hours to deliver and two people now understand the same code base, so you're covered if one of them wanders in front of a bus.  Everyone lives happily ever after.