Probably the HBH was overdone. Go light on it as the Pheromone in the Lemon Grass Oil is potent to a bee. To intense of an odor will force the bees out as if it were Beegone. The fact that you sprayed the inside of the hive with HBH tells me that you don't really understand its proper use. It has Lemon Grass Oil as an additive and is designed to be given to the bees via sugar syrup not used in lieu of a swarm lure.
Then, too, sometimes swarms already have their new home already chosen and nothing short of a queen includer is going to keep them from it. Sometimes they may wait several days, even draw some comb, but if they've already picked a new home say Bye Bye Bee!
If you can keep them in the box for a day, install a frame of brood to anchor the swarm. Not honey, that they will gorge themselves on and still take off anyway.