"He wouldn't even give me the time of day"

Okay so I was doing some deep thinking, as I do every once in a while, and you know the expression "He wouldn't even give me the time of day!" ? Do you think that maybe that expression began with someone asking someone else,"Hey what time is it?" and someone else being like "It's 7:00am" when it was really like 12. Or maybe they asked "What time of day is it" and the other guy was just like "Nah".

Yeah...