Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu write about experiments conducted on the Bing search engine:
It is of little value to extend experiments
that are statistically significantly negative after a couple of weeks.
Failing fast and moving on to the next idea is better.
Failing fast is a good thing: resources can be shifted to more promising ideas, instead of wasting more resources on an effort likely never to succeed as currently conceived.
Related to this idea is is the idea of being risk confrontational.
In any project, find the roughest problem in that project, the one most likely not to succeed among those that have to succeed for the project to work. This is the part of the project with the highest risk. That risk should be confronted up front, before the rest of the project details are worked out.
1. Try to solve that problem.
2. If you can’t solve it, try to find someone who can.
3. If steps 1 and 2 fail, give up.
Note this is generally what people and organizations don’t do. They start with the easy part of the project, often saying “we’ll figure out the rest somehow”. But big problems don’t tend to solve themselves, and when they are solved sometimes demand that the rest of the project be re-thought in order to allow the big problem to be solved.