To inspect the bottom box, do I remove the entire top box with frames in it?
you will probably need to pull those honey frames before you try to lift the box....unless you have a discount at your chiropractor :-) put them in another box temporarily AND COVER THEM or you may start robbing while you are trying to work.
it is very unlilkly that your queen will be on the honey frames, but you may smoke them a little to drive her and her friends down. then start at the outside and remove the frames.
If I put a queen excluder on top of the 2nd box now and add on another box of frames/foundation will I be able to extract some honey for ourselves, while allowing the bees to continue building strength for winter?
a couple of things here: i suspect that you are honey bound. this means that you may already be behind in brood production. 1st, get some of those honey frames out of the 2nd deep and replace then with drawn comb if you have it, or foundation if you don't. 2nd, i would not use an excluder. if you do not have drawn comb for your honey super, the excluder will be enough to discourage them from going up there and working it. then you are back to your orriginal problem. + you need to get brood production up if you are honey bound and if the queen chooses to go into the honey super, that's not the end of the world.
while you are checking out those bottom boxes look for swarm cells.