According to a recent LinkedIn post by Gary Illyes, Analyst at Google, you should be cautious if Google starts aggressively crawling your website.
While an uptick in crawling can be a good sign, Illyes says it may indicate underlying issues.
Illyes cautions:
“Don’t get happy prematurely when search engines unexpectedly start to crawl like crazy from your site.”
He says there are two common problems to watch out for: infinite spaces and website hacks.
Infinite Spaces Could Cause Crawling Spike
An issue Illyes highlighted is sites with “infinite spaces”—areas like calendar modules or endlessly filterable product listings that can generate unlimited potential URLs.
If a site is crawled a lot already, crawlers may get extra excited about infinite spaces.
Illyes explains:
“If your site generally has pages that search users find helpful, crawlers will get excited about these infinite spaces for a time.”
He recommends using the robots.txt file to block crawlers from accessing infinite spaces.
Hacked Sites Can Trigger Crawling Frenzy
Another troubling cause of a crawling spike is a security breach where hackers inject spam onto a reputable site.
Crawlers may initially interpret this as new content to index before realizing it’s malicious.
Illyes states:
“If a no-good-doer somehow managed to get access…they might flood your site with, well, crap… crawlers will get excited about these new pages for a time and happily crawl them.”
Remain Skeptical Of Crawling Spikes
Rather than assuming a crawling spike is positive, Illyes suggests treating it as a potential issue until the root cause is identified.
He states:
“Treat unexpected sharp increases in crawling as a symptom…until you can prove otherwise. Or, you know, maybe I’m just a hardline pessimist.”
Fixing Hacked Sites: Help From Google
For hacked sites, Illyes pointed to a page that includes a video with further assistance:
Here are the key points.
Tips From Google’s Video
Google’s video outlines the steps in the recovery process.
1. Identify The Vulnerability
The first crucial step is finding how the hacker gained access. Tools like Google’s Webmaster Tools can assist in detecting issues.
2. Fix The Vulnerability
Once the security hole is identified, it must be closed to prevent any future unauthorized access. This could involve updating software, changing passwords, etc.
3. Clean The Hacked Content
Check the entire site’s content and code to remove any spam, malware, defaced pages, or other injections by the hacker. Security plugins like Wordfence can assist in this process.
4. Harden Security
Beyond fixing the specific vulnerability, take additional measures to harden the site’s security. This could include enabling firewalls, limiting user permissions, and more frequent software updates.
5. Request A Review
Once the vulnerability is patched and any hacked content is removed, you can then request Google to review the site and remove any security warnings or blacklists once it’s verified as clean.
The video notes that the review process is faster for malware issues (days) than spam issues (weeks) since Google has to inspect spam cleanup efforts further.
Additional Tips From Google’s John Mueller
Google’s John Mueller has previously offered specific advice on recovering from the SEO impact of hacked pages:
- Use the URL removal tool to deindex the hacked pages quickly.
- Focus on improving the overall site quality beyond removing hacked content.
- Lingering impacts may persist for months until the site recovers Google’s trust.
Why SEJ Cares
Website security is crucial for all businesses, as hacked content can impact trust and search engine rankings.
Google’s Gary Illyes pointed out that sudden spikes in crawling activity could indicate security breaches or technical issues that need immediate attention.
Featured Image: Stacey Newman/Shutterstock