When I was running a site, I had special rules in my firewall to look for things that said they were googlebot but which didn’t come from one of googles published public IPs.
I had that thought after I replied when I realized that most of the reputable search crawlers will publish the IPs/ranges they use in addition to the UA. The disreputable ones (cough Bytedance cough Xiaomi cough) will just spoof Chrome on Windows 10 and flood you with requests from AWS datacenters in Shanghai or Singapore.
That said, I may still continue looking into working with one of the actual search engines (vs meta search) and see how well that works.
When I was running a site, I had special rules in my firewall to look for things that said they were googlebot but which didn’t come from one of googles published public IPs.
…yeah.
I had that thought after I replied when I realized that most of the reputable search crawlers will publish the IPs/ranges they use in addition to the UA. The disreputable ones (cough Bytedance cough Xiaomi cough) will just spoof Chrome on Windows 10 and flood you with requests from AWS datacenters in Shanghai or Singapore.
That said, I may still continue looking into working with one of the actual search engines (vs meta search) and see how well that works.