Robots.txt | ww.wp.xz.cn

Resolved lauratraveler
(@lauratraveler)

3 years, 2 months ago
Hi,

I have checked:
- Apply noindex to Media? – attachment
- Apply noindex to Categories? –category
- Apply noindex to Tags? – post_tag
- Apply noindex to Formats? – post_format
- Apply noindex to Author pages?
- Apply noindex to Date archives?
- Apply noindex to Search pages?
but my robots.txt only displays
```
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
```
Is this correct? Shouldn’t there be more to it?

Also, if I want to add:
```
User-agent: ChatGPT-User
Disallow: /
```
to my robots.txt. How can I do this?

Viewing 3 replies - 1 through 3 (of 3 total)

Plugin Author Sybre Waaijer
(@cybr)

3 years, 2 months ago

Hello!

That is correct. WordPress is dynamic, so we cannot predict where attachments, categories, or post tags may be.

We use robots meta tags to prevent search engines from indexing those pages, as Google explains here: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag.

If you wish to edit the robots.txt output, please also use https://ww.wp.xz.cn/plugins/robots-txt-quick-editor/.
Thread Starter lauratraveler
(@lauratraveler)

3 years, 2 months ago
Thanks @cybr,

But I’d rather not add yet another plugin. Is adding a robotx.txt file with the following:
```
User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /search/
Allow: /wp-admin/admin-ajax.php

User-agent: ChatGPT-User
Disallow: /
```
under the root folder of my domain a good solution?

Will the noindex tag still be applied to media, attachments, categories, etc?

Thank you!
Plugin Author Sybre Waaijer
(@cybr)

3 years, 2 months ago
Howdy!

Yes, that would work for most sites.

I recommend adding a sitemap: part too, as you can see here: https://theseoframework.com/robots.txt. That helps some search engines locate the sitemap automatically.

The robots meta settings of TSF work independently from the robots.txt output; so yes, those still work.
- This reply was modified 3 years, 2 months ago by Sybre Waaijer. Reason: Answered second question

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Robots.txt’ is closed to new replies.

3 replies
2 participants
Last reply from: Sybre Waaijer
Last activity: 3 years, 2 months ago
Status: resolved