query_posts pagination count

I have query_posts call on my main loop that is filtering post by category. I am only outputting any attached images to the post in the loop using a plugin that makes it easy to do with a template tag. So the loop just outputs a simple gallery composed of any images attached to a post in the right category.

Here is the issue.

Not all post have an image attached. They are still in the right category so they are counted toward the total number of pages displayed in the pagination, but no posts are shown. So I end up with a few extra blank pages.

Any way to make the pagination make an accurate count for what I want?