Is this a bug/inconsistency or operator error?

I am trying to remove some ajax actions that a plugin adds but for ajax actions, once they are added they cannot be removed.

I went round in circles for a while then tried just doing this with a test function:

add_action('wp_ajax_nopriv_ajax_test', array(&$this, 'ajax_test'));
        add_action('wp_ajax_ajax_test', array(&$this, 'ajax_test'));
        remove_all_actions('wp_ajax_nopriv_ajax_test');
        remove_all_actions('wp_ajax_ajax_test');

This test function is available through javascript on front end even though it is removed immediately after its added.

I realise that the ajax actions must work differently as there is no hard coded do_action. I tried stepping through the code of add_action but I could not see anything different.

I really need to be able to disable certain plugins ajax hooks but I cannot find a way.

Am I doing something wrong or is this a bug?