I too had the same issue. admin-ajax.php took around 3 seconds to load. So I disabled the file. So, far I noticed nothing bad.
@mayasi – Probably too late, but definitely DO NOT simply disable admin-ajax.php. If you disable admin-ajax.php more likely than not your checkout will fail.
