I noticed the hive isn't even attempting to defend themselves not even against me, just ignoring me all together no matter what I do.
Yes they are probably overwhelmed with the pests they are dealing with plus they have little to defend in terms of brood it sounds like. You said you found an inch long worm in there and normally they are able to remove them long before they get that size. Seeing one that size tells me the colony is near collapse.
I removed the infested frames and left the what look to the naked eye good frames.
The key phrase you used was "to the naked eye". I would freeze every
drawn frame in there as there may be eggs present which go unseen by the naked eye. You don't want to unknowingly transfer your problem to your other hive.
From what I was reading that if you get wax moth's most likely you are queenless. Is there any truth to that cause with the way they are acting and the numbers being terribly small (all I saw was honey in the brood chamber never saw a queen or young) no honey in the super.
Most likely they are queenless but I would still check for one. You might have a virgin queen who hasn't started laying yet and that could be trouble if you try to combine in that situation.
Personally I would combine as soon as possible because things can go quick once it reaches this stage.
I highly recommend the West SHB trap which I got from Dadant. I was only seeing 3-5 adults when I would inspect my hive1 (I have two hives also) but when I pulled the tray after five days guess how many dead ones were in the tray?
So if you are seeing only a few when you inspect, you can multiple that number several times to get a true picture of the problem.