Tile URL Crawled

  • Resolved Oliver Campion

    (@domainsupport)


    3 years, 8 months ago

    Hi,

    Your plugin injects the URL of the tile server into the <script> tag at the bottom of where the shortcode is placed …

    var baseUrl = 'https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png';

    We are using our own tile server URL which you can set in the settings …

    var baseUrl = 'https://example.com/wp-admin/admin-ajax.php?action=leaflet_js_tile&subdomain={s}&z={z}&x={x}&y={y}';

    Which works perfectly. Awesome.

    The problem is that Google (and possibly other search engines) are extracting the URL from the JavaScript and attempting to index it. This in turn is resulting in the following error from Google Search Console …

    Blocked due to other 4xx issue
    These pages aren’t indexed or served on Google
    Examples:
    https://example.com/wp-admin/admin-ajax.php?action=leaflet_js_tile&subdomain={s}&z={z}&x={x}&y={y}

    Is there a way that you can pass this URL to JavaScript without it being picked up by the search engines as a URL to crawl?

    Thank you in advance.

    Oliver

Viewing 8 replies - 1 through 8 (of 8 total)
  • Plugin Contributor hupe13

    (@hupe13)

    3 years, 8 months ago

    I would try this in /wp-admin/.htaccess:

    <Files admin-ajax.php>
	Header set X-Robots-Tag "noindex, nofollow"
</Files>

    But whether the search engines respect this is their business …

    Thread Starter Oliver Campion

    (@domainsupport)

    3 years, 8 months ago

    Pretty sure Google is ignoring the “noindex” header because failed ajax (which this presents) already has it …

    > --------------------------------------------
> 403 Forbidden
> --------------------------------------------

Status:	403 Forbidden
Code:	403
Date:	Wed, 05 Oct 2022 20:13:11 GMT
Content-Type:	text/html; charset=UTF-8
Connection:	close
X-Robots-Tag:	noindex
X-Content-Type-Options:	nosniff

    So that’s why I’m hoping that there’s a solution which removes the URL from being crawled in the first place … 😬

    Oliver

    Plugin Contributor hupe13

    (@hupe13)

    3 years, 8 months ago

    “noindex” means, Google crawls but doesn’t index it.

    Try “nofollow”. Google accepts this, but Bing not. Or you write disallow /wp-content/wp-admin/ in your robots.txt. But this is also just a recommendation for the search engines, some comply, some do not.

    The baseUrl is a variable parameter (option). When I programmed my plugin, I also found no solution how to provide such parameters other than inline.

    Thread Starter Oliver Campion

    (@domainsupport)

    3 years, 8 months ago

    Surely “nofollow” means search engines won’t follow the link to crawl further. If this is in a header then it’s a bit late by then?

    Getting a bit off topic here but whenever I develop a WordPress shortcode that requires JavaScript I use wp_enqueue_script() and wp_localize_script() to insert the .js file in the footer and pass dynamic variables.

    Anyway, if the plugin developer can think of a way to pass the baseUrl to the JavaScript in a way that won’t be read and crawled by search engines (maybe by base64 encoding and decoding it perhaps), that would be brilliant.

    Oliver

    Thread Starter Oliver Campion

    (@domainsupport)

    3 years, 5 months ago

    If any of the developers of this plugin read this, please can they incorporate this in their next update so that search engines stop indexing the baseUrl?

    Change line 317 of /shortcodes/class.map-shortcode.php from ..

    var baseUrl = '<?php echo filter_var($tileurl, FILTER_SANITIZE_URL); ?>';

    … to …

    var baseUrl = atob('<?php echo base64_encode(filter_var($tileurl, FILTER_SANITIZE_URL)); ?>');

    Thank you,

    Oliver

    Plugin Author bozdoz

    (@bozdoz)

    3 years, 5 months ago

    It’s a cool idea. Extremely specific use-case. I’ll take a look.

    Referenced here:

    https://github.com/bozdoz/wp-plugin-leaflet-map/issues/187

    Thread Starter Oliver Campion

    (@domainsupport)

    3 years, 5 months ago

    Nice one! 🙂

    Plugin Author bozdoz

    (@bozdoz)

    3 years, 5 months ago

    Fixed in 3.1.0

Viewing 8 replies - 1 through 8 (of 8 total)

The topic ‘Tile URL Crawled’ is closed to new replies.