um_prepare_user_query_args not working

I see no changes when applying this filter for example in my child theme functions.php

function um_custom_prepare_user_query_args( $query_args, $directory_settings ) {

	$query_args['exclude'] = array( 3, 4 );
	return $query_args;
}

add_filter( 'um_prepare_user_query_args' ,"um_custom_prepare_user_query_args", 10, 2 );

I still see users with ID 3 and 4 in the directory…I’ve also tried other query modifications and they don’t seem to do anything. Such as:

add_filter( 'um_prepare_user_query_args' ,"um_custom_prepare_user_query_args", 999999999, 2 );
function um_custom_prepare_user_query_args( $args, $directory_settings ) {
    $test_only_one_user = "[email protected]";

	$args['meta_query'] = array(
        "relation" => "OR",
        array(
          'key' => 'user_email',
      'value' => serialize( $test_only_one_user ),
      'compare' => 'LIKE'
        ),
        array(
      'key' => 'user_email',
      'value' => $test_only_one_user,
      'compare' =>  '='
        )
      );

	return $args;
}

still makes no difference.

Any ideas how I can filter and exclude users programmatically (and globally) within the directory results?