I wouldn't sweat it. The only thing I'd worry about is removed frames full of bees getting knocked over. If you don't have a frame rest, I'd get one. I just started using one I bought. It's wonderful! Before I had to either lean it up somewhere or hang on to it while messing with another one-handed. If you don't have any brood frames exposed to the wind for too long, it shouldn't chill the brood.