Ants love being out of the rain. Those little ones that like to move their larvae around, like to move it where they get heat from the sunshine, but no rain (like between the cover and the top bars or between a cover and an inner cover). I would just brush them off. if they have a big nest inside the actual box, I would just flip it upside down and dump them out. A good medium strength slam to the ground should get almost all of them.