Google’s Search Relations answered a number of questions concerning webpage indexing on the most recent episode of the ‘Search Off The Record’ podcast.
The matters mentioned have been learn how to block Googlebot from crawling particular sections of a web page and learn how to stop Googlebot from accessing a website altogether.
Google’s John Mueller and Gary Illyes answered the questions examined on this article.
Blocking Googlebot From Particular Internet Web page Sections
Mueller says it’s unattainable when requested learn how to cease Googlebot from crawling particular net web page sections, reminiscent of “also bought” areas on product pages.
“The short version is that you can’t block crawling of a specific section on an HTML page,” Mueller stated.
He went on to supply two potential methods for coping with the problem, neither of which, he careworn, are superb options.
Mueller recommended utilizing the data-nosnippet HTML attribute to stop textual content from showing in a search snippet.
He reassured everybody listening that if the content material in query is being reused throughout a number of pages, it’s not an issue that wants fixing.
“There’s no need to block Googlebot from seeing that kind of duplication,” he added.
Blocking Googlebot From Accessing A Website
In response to a query about stopping Googlebot from accessing any a part of a website, Illyes supplied an easy-to-follow resolution.
“The simplest way is robots.txt: if you add a disallow: / for the Googlebot user agent, Googlebot will leave your site alone for as long you keep that rule there,” Illyes defined.
For these in search of a extra sturdy resolution, Illyes affords one other technique:
“If you want to block even network access, you’d need to create firewall rules that load our IP ranges into a deny rule,” he stated.
See Google’s official documentation for an inventory of Googlebot’s IP addresses.
Although it’s unattainable to stop Googlebot from accessing particular sections of an HTML web page, strategies reminiscent of utilizing the data-nosnippet attribute can supply management.
When contemplating blocking Googlebot out of your website completely, a easy disallow rule in your robots.txt file will do the trick. However, extra excessive measures like creating particular firewall guidelines are additionally out there.
Featured picture generated by the writer utilizing Midjourney.
Supply: Google Search Off The Record