do you see dried up grayish things on the bottom board or being pulled out.
http://www.google.com/search?tbm=isch&hl=en&source=hp&biw=1680&bih=881&btnG=Search+Images&gbv=2&aq=f&aqi=g2&oq=&q=chalkbroodif it is chalkbrood, it can slow down a hive and in rare cases, kill it. requeening is often recommended, but i have not found it to be useful. chalkbrood is caused by a fungus and so is usually self limiting. when the conditions that are favorable for this fungal growth are gone, the chalkbrood disappears ....but you may see it year after year.
i hear that you guys are also having weather issues that are making forage scarce? you might try feeding and see if you can stimulate that queen.
Could it just be neglected brood dieing off because of the population drop in the hive.
yes....is most of it together and in places where the bees might not have spread to tend the brood?