Think about the negative feedback mechanism and homeostasis. If they are constantly flipping the thermostat up and down, their bodies may not have the ability to regulate and it ruins their balance. With the elderly this is even more the case. They tend to be more cold natured anyway because physiologically their skin is thinner and they lose heat a lot faster. Sort of the same with babies.