Yes, those are queen supercedure cells. The bees will make those sometimes, I don't know why. What is doubly puzzling about those things is that they often will draw them out over a full pollen cell.
That being said, it isn't uncommon for bees to supercede a queen, especially after a traumatic experience like being stuck for days on end in this tiny cage with nowhere to fly or poop and only sugar water to drink. If that is the case...you can try to stop them but I don't think that the old queen will get killed until after the new queen hatches so it should be ok.
I wouldn't worry about it too much. If they do start raising a youngster in there...just let them be.