I abandoned trying to link related posts in the content editor because – as you discovered – I got the feeling that it was an issue with core.
I worked with the designer to instead add related links in a sidebar using http://ww.wp.xz.cn/support/view/plugin-reviews/related-links which does include custom post-types.
Here’s a demo on our temporary dev site: http://dev.puppeteers.org/puppetry-journal/journals/spring-2011-volume-62-no-3/
The “In this journal” section is created using the Related Links plugin.
I’m guessing the way to track this down will be through TinyMCE and how that particular button queries the database.
With the first option I’m looking into adding the shortcode_unautop to my shortcode function.
function col1_shortcode($atts = null, $content = null){
$content = trim( do_shortcode( shortcode_unautop( $content ) ) );
$col = '<div class="col1">' . $content . '</div>';
return $col;
}
We’ll see if that works.