noindex issues

Resolved highlandmoss
(@highlandmoss)

2 years, 5 months ago
I have spent some time over the last few days trying to debug this issue. I have a lot of pages that are not currently indexed due to a noindex meta tag.

I have narrowed it down to SEO Framework (disabling the plugin resolved the issue)

I checked all SEO Framework settings as well as settings in any other plugins I have which could theoretically have some kind of setting. I also checked the affected pages (in this example it is a product category) and ensured they are set to be indexed.

Here is the complaint from Google Search Console:

——

Indexing allowed?

error

No: ‘noindex’ detected in ‘robots’ meta tag

——

This is a sample url that is failing:

https://highlandmoss.com/house-plants/wee-plants/

Here is the meta tag being generated:
```
<meta name="robots" content="noindex,max-snippet:-1,max-image-preview:large,max-video-preview:-1" />
```
I attempted to manually override the issue by adding this code snippet, but it has no effect.
```
function remove_noindex() {
    // Check if it's a single post of type 'post'
    if ( is_single() && get_post_type() == 'post' ) {
        echo '<meta name="robots" content="index,follow" />';
    }
    // Check if it's a WooCommerce product category
    elseif ( is_product_category() ) {
        echo '<meta name="robots" content="index,follow" />';
    }
}

add_action( 'wp_head', 'remove_noindex' );
```
I flushed the object cache, wprocket, cloudflare etc but now running out of ideas.

As SEO Framework is made with an abundance of confidence, it does mean a lot of things are automated and so I don’t want to tinker with it. I was hoping for direct help from the developer but this forum is where I was directed.

I did a search and found other people had sort of similar issues but couldn’t find a solution that worked for my instance.

Thank you to anyone that can help me out.
- This topic was modified 2 years, 5 months ago by highlandmoss.
The page I need help with: [log in to see the link]

Viewing 15 replies - 1 through 15 (of 21 total)

1 2 →

Plugin Author Sybre Waaijer
(@cybr)

2 years, 5 months ago

Hello! I’m the plugin developer, helping out full-time in these forums.

Please note that the most restrictive robots directive is in effect. So, “index” won’t override “noindex”.

It appears you may have checked a “noindex” in TSF’s settings.

Please go to “SEO Settings > Robots Meta Settings > Indexing” and uncheck product_cat.

It should look a bit like this: https://i.imgur.com/w4g0iHA.png.
After updating, please flush the site’s caches.

Let me know if this resolves the issue. If not, there are other more apt filters to force indexing of the archives, but I think it’s best to eliminate the cause of the issue so you can still safely rely on the automated “noindex” feature.

Thread Starter highlandmoss
(@highlandmoss)

2 years, 4 months ago

Hi,

Thank you for your reply. Unfortunately, as I said, I have combed through every SEO Framework setting and I don’t have a single thing checked as ‘noindex’.

Seo Framework settings
Plugin Author Sybre Waaijer
(@cybr)

2 years, 4 months ago
Thanks for that image; it literally paints a clearer picture.

Could you add this to wp-config.php?
```
define( 'THE_SEO_FRAMEWORK_DEBUG', ( $_GET['tsf_debug'] ?? false ) === 'true' );
```
Then, if we access any of your pages with ?tsf_debug=true attached, we can inspect how TSF determined the query near the bottom of the pages. No sensitive information will be made available, but I recommend removing it after I’ve reviewed it (I’ll post feedback here).

The debugging window will look like this:

After I’ve collected this info, I can pursue a fix or provide a (temporary) workaround.

While we’re trying to figure out what’s wrong (I’m assuming Kadence Blocks is breaking the query), I recommend keeping the Media, Formats, Date, and Search checked; their checkmarks are green, indicating that they are checked by default. This will prevent Google from indexing a bunch of nonsense, and it won’t affect the actual pages of your site.
erika998
(@erika998)

2 years, 4 months ago

I think we are having a similar issue. After installing the SEO Framework our site is has a noindex also listed. We also have no noindex options checked. Our site url: laaconline.org

The text:

head > meta

<meta name=”robots” content=”noindex, nofollow” />

Here is the pagespeed report.
https://pagespeed.web.dev/analysis/https-www-laaconline-org/xo891in6km?form_factor=desktop

Thread Starter highlandmoss
(@highlandmoss)

2 years, 4 months ago

Hi,

Thank you for the updated information.

I have added the code to wpconfig and disabled indexing on the things you suggested.

The only one I left off was search. I use Searchanise and I have modified it slightly to replace my woocommerce shop loop. So I fear if I was to noindex searches, it might consider my whole shop as search results.
Plugin Author Sybre Waaijer
(@cybr)

2 years, 4 months ago
Thank you for following up.

Yes, it is an unconventional method to manage a website; it’s a hacky way for custom queries. I recommend sticking to product categories or custom taxonomies in the future.

Still, I tried the debugging parameter, and it appears to be inactive.

Could you confirm you’ve included the definition in wp-config.php? Perhaps your modification got blocked during upload due to strict file permissions, which you may need to make more lenient temporarily as you edit the file. To learn more, see https://developer.ww.wp.xz.cn/advanced-administration/server/file-permissions/.

I’m afraid I cannot comment on the “search” aspect of your site; this is exactly why I wanted to inspect how TSF assesses the queries: I need to investigate why a taxonomy request returns a search result.

@erika998 I couldn’t find that issue on your site. If you’re still facing this issue, please open a new support topic. Thanks!
- This reply was modified 2 years, 4 months ago by Sybre Waaijer. Reason: more info
Thread Starter highlandmoss
(@highlandmoss)

2 years, 4 months ago

I double checked wp-config now and downloaded the file again after uploading and can see the code in the file.

I moved the code to the top (but within php tags). Can you see if it is working now?

View post on imgur.com
Plugin Author Sybre Waaijer
(@cybr)

2 years, 4 months ago
Thank you! It works now.

Your Product Categories have products, but a plugin sets the Product count to 0 (probably to fault the query and hijack the page more easily). So, TSF thinks the term isn’t populated and will return a 404 response.

This simple filter should fix the issue for your website:
```
add_filter( 'the_seo_framework_enable_noindex_no_posts', '__return_false' );
```
I recommend clearing your site’s caches after adding the filter.

If you’d like me to verify if it’s working as intended, leave the debugging line in place, and I’ll inspect a few pages.

But before that, I still recommend applying “noindex” as aforementioned: at “SEO Settings > Robots Meta Settings > Indexing,” keep the Media, Formats, Date, and Search checkboxes checked.

The “noindex” for “Search” will not affect your custom search pages. Here’s why:
1. The categories showing search results are not interpreted by WordPress (or TSF) as Search but as they were intended: Product Categories.
2. When a user enters a custom search query, the advanced search results page shown is recognized as a Page. This is fine since TSF points the canonical URL to the search page root (populated with products), so your site’s index at Google won’t be tainted by infinite user-generated search result pages.
The original WordPress Search endpoint still works (/search/* and ?s=*), but you’ve opted for the more advanced search system. Therefore, I recommend unchecking “Add Sitelinks Search Box” at “SEO Settings > Schema.org Settings > General,” for that option points to the unutilized search field.

Alternatively, you could filter the Schema.org output, but this adds more work than it’s worth (I still haven’t found a single site where Sitelinks Search Box was effective, regardless of it being a Google standard).
Thread Starter highlandmoss
(@highlandmoss)

2 years, 4 months ago

Thank you so much for this. Brilliant support and I appreciate you explaning not only how it works, but how my specific (janky as it may be) setup works in relation to your plugin and general SEO.

I know using Searchanise to replace my shop loop is a bit hacky but I have been through dozens and dozens of different plugins and ways of presenting my shop and have found this was best for performance and user experience.

Eventually I will replace this again with something more custom that utilises the native features.

I just added the snippet, cleared the cache and tested a live url and got the same noindex error. I will do a deeper clearout from cloudflare and on my hosting panel (maybe object cache didnt flush properly for example) and try again.

If you are around today I would appreciate you giving it a final check.

I noindexed search results as suggested – I already had followed your instructions on the others. I have some other items in the list from hooked elements but will come back to that once we have this indexing issue sorted first.

Thanks again!

Thread Starter highlandmoss
(@highlandmoss)

2 years, 4 months ago

Update: Just did two more tests on category pages and they are now indexing.

I have loads of links like ?addtobasket queries in the noindex list – is this an issue? I know I don’t want them indexed but I would rather junk stuff like this wasn’t crawled in the first place as it makes it hard to see what is working properly.
Plugin Author Sybre Waaijer
(@cybr)

2 years, 4 months ago
Hello!

The ?add-to-cart and similar links must have “noindex,” for they contain no valuable content to the search engine or new visitors landing on your site.

You cannot really prevent crawling of this; crawlers are programmed to touch anything that even remotely looks like a link. This is where crawl-control comes in, like “noindex,” “nofollow,” robots.txt, or downright blocking them via a firewall. Unless your server experiences downtime from crawling or slow indexing performance, “noindex” is enough.

I took another gander, and almost everything appeared to be okay.

However, the search results pagination uses ?page instead of ?paged, which causes pagination detection to fail at some points. This is because paged is used for singular pages and page for real archives. Since the Searchanise result pages aren’t presented as archives but as singular pages, the paginated URLs cannot be generated correctly for them. So, all canonical URLs on /shop-2?... pages will point to /shop-2/. This can prevent deep crawling of the archive, and so search engines can miss old products.

You mitigated this by creating landing pages for the different moss product types, like /made-with-moss/moss-poles/, /house-plants/, /christmas-range/, etc. Everything works well as long as /shop-2? isn’t the URL.

Still, to mitigate this further, I recommend disabling the optimized sitemap at “SEO Settings > Sitemap Settings,” setting the Query Limit to 200 or 300, and using /wp-sitemap.xml. You may need to resave the Permalink settings of WordPress for that sitemap to appear. This sitemap will contain a complete history of all indexable products, which makes search engines rely less on crawling your site deeply. You should submit this new sitemap to Google Search Console.

If Searchanise switches from ?page to ?paged, or if it could provide an option to switch between the two, this would be better. Hijacking the WordPress query is never a good idea and comes with many headaches — it’s why I am advocating to remove the Query Loop Block from WordPress.

I also still recommend unchecking “Add Sitelinks Search Box” at “SEO Settings > Schema.org Settings > General,” for that option’s output points to an unutilized search field.
- This reply was modified 2 years, 4 months ago by Sybre Waaijer. Reason: clarity
- This reply was modified 2 years, 4 months ago by Sybre Waaijer. Reason: additional info
Thread Starter highlandmoss
(@highlandmoss)

2 years, 4 months ago

Thank you so much for this, very informative and excellent support.

I unchecked sitelinks and I have approached Searchanise about your suggestions (I have a plan with them where they will customise the plugin for me).

This is what they responded with (I gave them the link to this thread and quote you directly):

“

Hello Craig,

Thank you for your message.

Do I get it right that you’d like us to change URLs like https://highlandmoss.com/shop-2/?page=1&rb_custom_taxonomy_plant-genus=Begonia to https://highlandmoss.com/shop-2/?paged=1&rb_custom_taxonomy_plant-genus=Begonia ?

Please confirm or elaborate on your query. Thank you!”

I let them know that this particular level of SEO and understanding pagination is beyond my knowledge but I think this is what you meant.

Can you please confirm if this is how the links should be structured? I would prefer this method as you suggested it was better than using a default sitemap.

Thank you!

Plugin Author Sybre Waaijer
(@cybr)

2 years, 4 months ago

I’m happy to help 🙂

Yes, that is what I meant — their support understood you well. This way, WordPress and TSF will know that pagination is engaged for the request. Still, I’d like to verify if this is the case after implementing the change.

For your site, I still recommend using the default sitemap to mitigate any mistakes with the site’s structure — TSF augments it so sitemap errors pertaining to indexability shouldn’t occur with Google Search Console.

Thread Starter highlandmoss
(@highlandmoss)

2 years, 4 months ago

Hi Sybre, I got an update from Searchanise but I am not sure how to check if their fix is effective or works as you suggested, would you mind having a quick look?

The fact is that the page parameter is used in our widgets code in different places, which is why changing it requires rewriting a significant part of the core’s code on our server’s side. Unfortunately, at the moment we cannot fulfill this request, because, technically, we are talking about changing the search logic for this parameter.

However, for our part, we have added a modification that replaces the parameter with paged directly in the URL. The difference is that this happens not at the server request level, but at the frontend. Please check if this solves the SEO issue and kindly let us know.
Searchanise Support

Thread Starter highlandmoss
(@highlandmoss)

2 years, 4 months ago

I really need to hear from you as I saw a sudden drop in sales (other than from local customers) which made me think we had an indexing issue.

When I checked, it looked like everything was indexed and all good – but then when I started checking URLs I could see pretty much everything except our main page is no longer indexed!

I am happy to ask searchanise to undo the changes they made but I would really like you to see if you can see the cause for sure please.