Diabling wordpress for a single directory

I’ve got a site where I’d like to disable wordpress from giving me a 404 error for my control panel URL.

Usually I can use http://www.domain.com/ControlPanel to get to the control panel, but not when wordpress is installed. Is there a way around this?