There's little-to-no relationship between flaccid and erect size. When I'm swimming in cold water, it sucks up inside and my balls tuck so tight that they're basically invisible. For all intents and purposes, I look like a girl in cold water. Well, down there, anyways! When it's comfortably warm, everything just kinda hangs out.
You might have a slightly lower than normal body temperature, so they'd suck tight to keep the temperature optimal when not in use. That or you keep your home at a temperature that causes it to suck in.
Note: I'm an alien (from Phoenix), so my definition of 'cold' and 'warm' is skewed compared to the rest of humanity. I consider an 85F room temp to be 'comfy', but most people would be sweating at that range.