When you operate with bees like that in late afternoon or in the evening, is normal that bees do not move inside. They stay there where you shaked them. They are like sleepy.
Often I cover them with newspaper to give them heat shelter. They often move inside but they may remain there over night.