Empty Category Base

Since I have my permalinks as:
/archives/%year%/%monthnum%/%postname%/

I would like to have my category base as ‘/archives’ like: http://www.mysite.com/blog/archives/categoryname

If I do this, the categories behave the way I want it, but the permalinks of the posts and everything else under /archives become 404s. Is there any way to get around this?