In terms of scientific / statistical investigation, it's interesting to contrast the following two principles:
Epicurus' principle of multiple explanations ("Keep all hypotheses that are consistent with the facts!"),
William of Occam's Razor ("Use the simplest explanation possible")
These would seem to be in opposition: do we want complexity, or simplicity?
I'm reminded of my first year graduate seminar, organized by Jack Atkinson at the University of Michigan. One week, Jack told us the world was a complex place and therefore complex models were needed to understand it.
The next week, Bob Zajonc came in and told us that we should try to find simple phenomenon that explained a lot and could be extended. [Bob, for example, came up with simple, extendable explanations on social facilitatation, the effect of mere exposure making people more positive, and birth order]
Over time, I've come to believe that Bob had the better of the argument. At heart, the world is full of relatively simple phenomenon that produce a more complex world via relatively simple rules. For example, there are only a few different molecules in DNA, but they can produce complex organisms.
But, in the course of discovering these phenomena in real data, you need to avoid prematurely coming to a conclusion. That's where Epicurus comes in.