Custom permalink for posts and not portfolio items

I have searched this on google and not really came up with any solution, unless I missed something or didn’t search correctly

My current website is setup with a wordpress blog at http://www.domain.com/blog/

I am in the process of creating a new website that everything is handled by wordpress including the portfolio section. My issue is all the blog posts I have which include facebook integration all tied to the URL are all under the http://www.domain.com/blog/ url.

I’ve tried changing the custom permalinks to include /blog/ at the begining but that for some reason adds it to the portfolio link as well so it comes out http://www.domain.com/blog/portfolio/%postname%/

Current permalink setting is: /%year%/%monthnum%/%day%/%postname%/

So given that the permalink structure is different than the portfolio URL structure why does addding this to the permalink: /blog/%year%/%monthnum%/%day%/%postname%/ cause me to get blog infront of the portfolio URL and how can I get this structure below

Anything under posts: http://www.domain.com/blog/%year%/%monthnum%/%day%/%postname%/

Anything under portfolio: http://www.domain.com/portfolio/%postname%/ (as it is currently)

Currently pots are going to: http://www.domain.com/%year%/%monthnum%/%day%/%postname%/

The blog home page is at: http://www.domain.com/blog/ using a custom template on a page.

Thanks in advance for the help and hope this all makes sense.