Ask Question Forum:
Model Library:2025-02-08 Updated:A.I. model is online for auto reply question page
C
O
M
P
U
T
E
R
2
8
Show
#
ASK
RECENT
←
- Underline
- Bold
- Italic
- Indent
- Step
- Bullet
- Quote
- Cut
- Copy
- Paste
- Table
- Spelling
- Find & Replace
- Undo
- Redo
- Link
- Attach
- Clear
- Code
Below area will not be traslated by Google,you can input code or other languages
Hint:If find spelling error, You need to correct it,1 by 1 or ignore it (code area won't be checked).
X-position of the mouse cursor
Y-position of the mouse cursor
Y-position of the mouse cursor
Testcursor
caretPos
Attachment:===
Asked by test test
at 2025-05-05 20:08:11
ModifyPoint:500 Replies:3 POST_ID:829366USER_ID:12424
Topic:
googlebot;crawler;cloudflare
How my website can use cloudflare with"Under Attack Mode" and be able to allow goolge bot, crawler to crawl my website pages at the same time ?
Please advise
Author: test test replied at 2025-05-05 20:25:59
And this is NOT working from set up customer rules for Javascript Challenge if AS numver is not question to google one at
Accepted Solution
Author: test test replied at 2025-05-05 20:23:18
500 points Excellent
Cloudflare console, at security->WAF->Tools->IP Access rules,: add value, AS15169, allow,select all website in account. AS number u can find at WAP-->event
https://community.cloudflare.com/t/allow-google-bot-with-rules/504109 Expert: rcschroeder replied at 2025-05-05 20:10:41
I'd be happy to help you understand why Googlebot (also known as a crawler or spider) may not be able to crawl your website when Cloudflare is in "Under Attack Mode" and provide guidance on how to allow Googlebot to crawl while still using Cloudflare's security features.
**Why doesn't Googlebot crawl my website when Cloudflare is in Under Attack Mode?**
Cloudflare's Under Attack Mode (UAM) prioritizes protecting your website from incoming attacks, such as DDoS attacks. When UAM is enabled, it blocks all non-essential traffic, including crawlers like Googlebot. This includes blocking the requests sent by Googlebot to crawl your website.
**Can I allow Googlebot to crawl while still using Cloudflare's Under Attack Mode?**
Yes, you can configure Cloudflare to allow Googlebot to crawl while still protecting your website from attacks. Here are a few methods:
1. **Configure the allowed crawl agents**: In Cloudflare, go to "Security" > "Crawling", and then select "Allowed Crawlers". Add `googlebot` (or `Googlebot/2.1` or `Googlebot-Mobile/2.1`) to the list of allowed crawlers.
2. **Use a custom crawl rule**: Create a custom crawl rule that specifically allows Googlebot to crawl your website. You can do this by going to "Security" > "Crawling" and then clicking on "Rules". Click "Add Rule" and select "Custom Crawl Rule". In the rule, add `googlebot` as the crawler ID.
3. **Use a Cloudflare Proxied Origin**: This method requires some technical expertise. You'll need to configure your website's server to use a proxied origin with Cloudflare. This will allow Googlebot to crawl your website while still using Cloudflare's protection features.
**Additional tips:**
* Make sure you're using the latest version of Cloudflare and that you've enabled the "Crawling" feature in your account.
* If you're experiencing issues, try disabling UAM temporarily to see if Googlebot can crawl your website.
* Consider implementing a more advanced crawling solution, such as using Cloudflare's "Crawl Queue", which allows you to manage crawl requests and schedule them for specific times.
By following these steps, you should be able to allow Googlebot to crawl your website while still enjoying the security benefits of Cloudflare's Under Attack Mode.
**Why doesn't Googlebot crawl my website when Cloudflare is in Under Attack Mode?**
Cloudflare's Under Attack Mode (UAM) prioritizes protecting your website from incoming attacks, such as DDoS attacks. When UAM is enabled, it blocks all non-essential traffic, including crawlers like Googlebot. This includes blocking the requests sent by Googlebot to crawl your website.
**Can I allow Googlebot to crawl while still using Cloudflare's Under Attack Mode?**
Yes, you can configure Cloudflare to allow Googlebot to crawl while still protecting your website from attacks. Here are a few methods:
1. **Configure the allowed crawl agents**: In Cloudflare, go to "Security" > "Crawling", and then select "Allowed Crawlers". Add `googlebot` (or `Googlebot/2.1` or `Googlebot-Mobile/2.1`) to the list of allowed crawlers.
2. **Use a custom crawl rule**: Create a custom crawl rule that specifically allows Googlebot to crawl your website. You can do this by going to "Security" > "Crawling" and then clicking on "Rules". Click "Add Rule" and select "Custom Crawl Rule". In the rule, add `googlebot` as the crawler ID.
3. **Use a Cloudflare Proxied Origin**: This method requires some technical expertise. You'll need to configure your website's server to use a proxied origin with Cloudflare. This will allow Googlebot to crawl your website while still using Cloudflare's protection features.
**Additional tips:**
* Make sure you're using the latest version of Cloudflare and that you've enabled the "Crawling" feature in your account.
* If you're experiencing issues, try disabling UAM temporarily to see if Googlebot can crawl your website.
* Consider implementing a more advanced crawling solution, such as using Cloudflare's "Crawl Queue", which allows you to manage crawl requests and schedule them for specific times.
By following these steps, you should be able to allow Googlebot to crawl your website while still enjoying the security benefits of Cloudflare's Under Attack Mode.
