.Google's John Mueller answered an inquiry regarding why Google.com indexes webpages that are actually refused from crawling through robots.txt and why the it is actually secure to neglect the associated Explore Console files concerning those crawls.Bot Traffic To Inquiry Parameter URLs.The individual inquiring the inquiry recorded that crawlers were actually generating hyperlinks to non-existent question specification Links (? q= xyz) to pages with noindex meta tags that are actually likewise blocked in robots.txt. What triggered the concern is that Google.com is creeping the web links to those webpages, getting blocked out by robots.txt (without noticing a noindex robots meta tag) then receiving shown up in Google Explore Console as "Indexed, though blocked out by robots.txt.".The individual asked the adhering to question:." But below's the big inquiry: why will Google index pages when they can't also view the material? What's the conveniences during that?".Google.com's John Mueller affirmed that if they can not crawl the page they can not view the noindex meta tag. He also creates an appealing acknowledgment of the web site: search driver, advising to overlook the outcomes because the "typical" individuals won't see those end results.He created:." Yes, you are actually proper: if our experts can't crawl the page, our company can not find the noindex. That said, if our experts can not crawl the webpages, after that there's certainly not a whole lot for our team to mark. Thus while you could view some of those pages along with a targeted internet site:- query, the ordinary user won't view all of them, so I would not bother it. Noindex is actually additionally alright (without robots.txt disallow), it only suggests the URLs will definitely find yourself being crept (and wind up in the Browse Console record for crawled/not indexed-- neither of these statuses cause issues to the rest of the internet site). The vital part is that you don't make them crawlable + indexable.".Takeaways:.1. Mueller's response validates the limitations being used the Website: hunt progressed search operator for diagnostic causes. Among those main reasons is since it is actually certainly not connected to the frequent hunt index, it is actually a different thing entirely.Google.com's John Mueller talked about the site hunt operator in 2021:." The brief response is actually that a website: query is not meant to be full, neither made use of for diagnostics purposes.An internet site concern is a specific sort of hunt that restricts the results to a particular website. It's primarily just words internet site, a digestive tract, and after that the website's domain name.This query confines the end results to a specific internet site. It's certainly not indicated to be a thorough assortment of all the webpages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is actually fine for these sort of circumstances where a crawler is actually connecting to non-existent webpages that are getting found out by Googlebot.3. Links with the noindex tag will definitely generate a "crawled/not catalogued" entry in Browse Console which those won't have an unfavorable effect on the remainder of the web site.Read the inquiry and address on LinkedIn:.Why will Google mark pages when they can not even view the web content?Featured Photo by Shutterstock/Krakenimages. com.