Probably the HBH was overdone. Go light on it as the Pheromone in the Lemon Grass Oil is potent to a bee. To intense of an odor will force the bees out as if it were Beegone. The fact that you sprayed the inside of the hive with HBH tells me that you don't really understand its proper use. It has Lemon Grass Oil as an additive and is designed to be given to the bees via sugar syrup not used in lieu of a swarm lure.
You are probably right about the HBH. I just thought it would be better then the lemon pledge some people suggested I use. The second time I put them in, I layed the HBH on extra thick. Hope it disperses quickly.
Then, too, sometimes swarms already have their new home already chosen and nothing short of a queen includer is going to keep them from it. Sometimes they may wait several days, even draw some comb, but if they've already picked a new home say Bye Bye Bee!
If you can keep them in the box for a day, install a frame of brood to anchor the swarm. Not honey, that they will gorge themselves on and still take off anyway.
I did put in some brood, so I hope that will anchor them. I am also miles from where I picked up the swarm, so I know thay don't have a home nearby picked out.