I found this nugget on the Straight Dope.
Lazarus Geiger, a naturalist, …. Based on his examination of Greek literature, the Vedic hymns, and other ancient writings, he claimed that man's color sense had developed only gradually, in fairly recent times. He thought that man had become aware of colors in the order that they appear in the spectrum, starting with the longest wavelengths. First, he said, people dimly realized that something was "colored," then they could distinguish black and red, then black and red plus yellow, then white, then green, and finally blue. (I realize white is not a "color" of the spectrum, but this is Geiger's theory, not mine.) He pointed out that "Democritus and the Pythagoreans assumed four fundamental colours, black, white, red, and yellow. … Nay, ancient writers (Cicero, Pliny, and Quintilian) state it as a positive fact that the Greek painters, down to the time of Alexander, employed only these four colours."
This is very reminiscent of a favorite Calvin and Hobbes Sunday cartoon.
CALVIN: That's really weird.
CALVIN'S DAD: Of course, but they turned colors like everything else in the '30s.
CALVIN: So why didn't old black and white photos turn color too?
CALVIN'S DAD: Because they were color pictures of black and white, remember?
(CUT TO: EXT. Tree limb, Calvin talking with Hobbes)
Text from http://www.everything2.com/title/The+world+was+black+and+white Written by Bill Watterson, © Universal Press Syndicate.