Problems with sitemap and attachments

I’m having some really werid problems with my sitemap and ‘/?attachment_id=XXXX’ links (where XXXX is a 4 digit number). I have a bunch of these in my sitemap generated by WordPress SEO by Yoast which I have not had problems with in the past. There are many, many /?attachment_id=XXXX links in the sitemap. Most of them work. Some of them are being detected 404 by webmaster tools. Here is the really weird part, 3 out of 4 of the 404’s detected work, but only if logged in. 1 out of 4 is 404 logged in or not. There are more 404’s in the sitemap with format “/?attachment_id=XXXX” than just what’s currently detected by WordPress that I found by spot testing but there are too many to test them all. It seems like most of the links work.

How can I fix this? Is this a Yoast WordPress SEO problem? A WordPress problem? A Hueman theme problem? I’m thinking it cannot be a Yoast WordPress SEO problem only because of the links that can be seen when logged in but not when logged out.

Thanks.