No ifs about it. They did just move in. If you can get a cone on quick, before they store food or have brood, they will be out within days, queen and all.
PS. If they had been there awhile, they would have closed off the holes to the interior and be using the outside exit only.