I’v had a similar issue — the add image dialog would load as a normal page instead of in a overlay/popup.
This was caused by a javascript error in a plugin called “HeadSpace 2”.
You can use the Web-Inspector in Safari/Chrome or Firebug in Firefox to find these javascript errors, and then disable the offending plugin.
I have the same problem.
I enter a search term e.g. ‘snowcrash’, the search indicator lights up for a couple of seconds, and then nothing happens.
I’m using WordPress version 2.8.1-beta2