Working in a proclaimed data-driven company like Amazon, we extensively try to measure everything with weblabs, A/B tests, and experiments. When I first joined my current team, one of the more senior devs pulled me aside and asserted “Jason, everything should be weblabbed. Everything.”
I love data. I personally have almost 700 SQL queries in my queue, which comes out to an average of almost 1 new SQL query a day at my tenure here at Amazon. One thing I’ve realized, though, is that data can paint a very misleading picture, especially when the test is an incomplete test.
There are some initiatives that require a critical mass before they can make a difference. I imagine, for example, that a weblab that introduced only an incomplete number of product images on a detail page was initially negative. Once customers became reliant on images, however, it likely became a huge positive.
There are other examples where holding hard negotiation standpoint with a vendor leads to a short-term negative but is paramount for the overall strategy. A weblab would capture the short-term negative but not find the long-term potential.
I really enjoyed this passage from Scott Burkun’s “A Year Without Pants.”
“Data can’t decide things for you. It can help you see things more clearly if captured carefully, but that’s not the same as deciding. Just as there is an advice paradox, there is a data paradox: no matter how much data you have, you still depend on your intuition for deciding how to interpret them and apply the data. … When a culture shifts too far into faith into data, people with great intuition leave. They’ll find employment where their judgement is valued rather than remain as an annoyance in some powerful equation marker’s report/ Making great things requires both intuition and logic, not a dominance of one over the other”
The point is, the growth chart has fluctuations up and down and weblabs can help us find the local maximum. Web labs alone, however, cannot be the sole method to find the global maximum, or the path to the truly highest opportunity.