Title: Blocked by robots, HTML parse error
Last modified: November 3, 2016
---
# Blocked by robots, HTML parse error
* Resolved [aamerkheraj](https://wordpress.org/support/users/aamerkheraj/)
* (@aamerkheraj)
* [9 years, 7 months ago](https://wordpress.org/support/topic/blocked-by-robots-html-parse-error/)
* Hi
* I’m getting the following errors but I don’t know how to fix them. Could you
please help?
===============================================================
========================= Custom Status Codes
* The Link Checker uses the following custom status codes:
* 598 – Blocked by robots: The Link Checker was not able to access the page because
the access was blocked by the robots exclusion protocol.
* 599 – HTML parse error: The HTML code of this page could not be parsed because
of an error in the code or because the page was larger than 50 MB.
* Please note that it is also possible that a website returns these status codes
and if this is the case, they probably have another meaning.
===============
=========================================================================
* Blocked by robots – What can I do to fix this?
* HTML parse error – I don’t see anything wrong with the HTML code. I do have some
parameters like target=_blank but that should cause errors.
* Thanks,
Aamer
Viewing 5 replies - 1 through 5 (of 5 total)
* Plugin Author [Marco Beierer](https://wordpress.org/support/users/mbsec/)
* (@mbsec)
* [9 years, 7 months ago](https://wordpress.org/support/topic/blocked-by-robots-html-parse-error/#post-8392902)
* Hi Aamer
* Thank you for using the Link Checker.
* The “blocked by robots” error means that the Link Checker is blocked in your
robots.txt file. This file is normally placed in the root directory of your site.
If you have never heard about the robots.txt, you could have a look at the [Wikipedia article](https://en.wikipedia.org/wiki/Robots_exclusion_standard).
* To solve the issue, you could allow the Link Checker to visit all pages by adding
the following two lines at the end of your robots.txt. Just make sure that an
empty line is between these lines and the current last line.
* ```
User-agent: MB-LinkChecker
Disallow:
```
* For the “HTML parse error” I would need to have a closer look. Could you send
me the link to the affected page?
* Best regards,
Marco
- This reply was modified 9 years, 7 months ago by [Marco Beierer](https://wordpress.org/support/users/mbsec/).
- This reply was modified 9 years, 7 months ago by [Marco Beierer](https://wordpress.org/support/users/mbsec/).
* Thread Starter [aamerkheraj](https://wordpress.org/support/users/aamerkheraj/)
* (@aamerkheraj)
* [9 years, 7 months ago](https://wordpress.org/support/topic/blocked-by-robots-html-parse-error/#post-8392935)
* I will look into the robots.txt file.
* HTML parse error – I don’t know which pages are affected. Is there any way to
check?
* Plugin Author [Marco Beierer](https://wordpress.org/support/users/mbsec/)
* (@mbsec)
* [9 years, 7 months ago](https://wordpress.org/support/topic/blocked-by-robots-html-parse-error/#post-8393031)
* Normally the affected pages are shown in the results table.
* Or do you just think you got these errors because they are shown in the “custom
status codes” section? If so, it may be a misunderstanding, because this section
is just for documentation and should help with the interpretation of the results.
It does not mean that you are affected by these errors.
* Have you already started the Link Checker once?
* Thread Starter [aamerkheraj](https://wordpress.org/support/users/aamerkheraj/)
* (@aamerkheraj)
* [9 years, 7 months ago](https://wordpress.org/support/topic/blocked-by-robots-html-parse-error/#post-8393170)
* Makes sense. I was just reading the documentation.
* Thanks for correcting me.
* Regards,
Aamer
* Plugin Author [Marco Beierer](https://wordpress.org/support/users/mbsec/)
* (@mbsec)
* [9 years, 7 months ago](https://wordpress.org/support/topic/blocked-by-robots-html-parse-error/#post-8393189)
* Thank you for your reply.
* I may improve the description of this section in one of the next versions to
make this clearer.
* Let me know if you have further questions or something does not work.
* Best regards,
Marco
Viewing 5 replies - 1 through 5 (of 5 total)
The topic ‘Blocked by robots, HTML parse error’ is closed to new replies.
* 
* [Link Checker Pro](https://wordpress.org/plugins/link-checker/)
* [Frequently Asked Questions](https://wordpress.org/plugins/link-checker/#faq)
* [Support Threads](https://wordpress.org/support/plugin/link-checker/)
* [Active Topics](https://wordpress.org/support/plugin/link-checker/active/)
* [Unresolved Topics](https://wordpress.org/support/plugin/link-checker/unresolved/)
* [Reviews](https://wordpress.org/support/plugin/link-checker/reviews/)
* 5 replies
* 2 participants
* Last reply from: [Marco Beierer](https://wordpress.org/support/users/mbsec/)
* Last activity: [9 years, 7 months ago](https://wordpress.org/support/topic/blocked-by-robots-html-parse-error/#post-8393189)
* Status: resolved