Not that long ago I had a conversation with @mattwynne, which led to the hastily sketched piece of paper, below. The diagram on the left (since redrawn and blogged about by @tooky) shows the relationship between End-to-end tests and Business-readable tests. Not all Business-readable tests need to be End-to-end and not all End-to-end tests need to be business readable.
The middle part of the sketch is my attempt to show the relative size of these sets of tests. There should be far more Business-readable tests than End-to-end tests. It also shows that most End-to-end tests are Business-readable because there are very few situations where purely technical concerns require anything broader than integration tests. The point is, where possible, test the domain model directly and only use End-to-end tests to verify correct ‘wiring up’ of the entire system.
The far right of the sketch attempts to relate the Venn diagram to the well known Testing Pyramid. Business-readable tests that hit the domain model directly map to the middle section of the pyramid – integration/component tests. Business-readable tests that hit the full stack map to the top of the pyramid. Not shown is where non-business-readable End-to-end tests should map.
At this point I’m going to re-imagine the Testing Pyramid as a Testing Iceberg (another product of conversations with @mattwynne). Those portions of the iceberg above the waterline are business readable, while those below are not. As you can see, in this diagram there are examples of all test types both above and below the readability waterline.
Now I can map non-business-readable End-to-end tests to the submerged system test portion of the iceberg, which is very small because most End-to-end tests should be business-readable. Some projects may have specific technical concerns that can only be validated using a fully deployed system, and that are of no interest to business people, but these will be few and far between.
You may be wondering why I consider the Testing Iceberg important? In my next blog I’m going to talk about lack of trust between developers, testers and business people and I think the Testing Iceberg will help us talk about how to address this. I’ll also explain why some of the tests that verify components in isolation (‘unit’ tests) are above the waterline.