I was able to work around this using wp_get_object_terms(). I still get the Error notice, but it is working well enough for my specific purpose.
A quick and dirty solution, add this to your functions.php
add_action('admin_head', 'remove_yoast_pub');
function remove_yoast_pub() {
echo '<style>
.misc-pub-section.yoast-seo-score{
display: none;
}
</style>';
}
