Drones gather in mating areas Drone Congregation Areas (DCA's). Someone on here recently said they were sort of nomadic and move from hive to hive and had no loyalty to one hive. But you have only one hive. I had never heard this before. Being no threat, no stinger etc. I can see it.
When your hive did have brood, you did have a mixture of drone and worker brood right? Not a huge lopsided amount of drone brood? I would stick with two hives of the same type --- JMO.