Googlebot crawls my page too oftenHow to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?Googlebot craw rate is too slow for a huge siteHow to control old indexed page crawling rateGooglebot sends several hundred requests in a minuteGoogleBot crawling a page up to 3,000 times per dayGoogleBot is crawling my datepicker to inifnityHow often do search bots typically visit a website?Stop googlebot crawling URL more than once?
Can a Horizon Walker ranger use the Distant Strike feature to teleport out of the Forcecage spell?
Has the 75 move draw rule ever been invoked in a FIDE rated standard game involving a GM?
Does no-one standing against the speaker of the house in UK lead to the local electorate being disenfranchised?
Help resolve territory acquisition design difference of opinion in MMO RTS
Who originated the dangerous avocado-pitting technique?
Impeachment jury tampering
Did the Byzantine Emperor Constantine XI will his titles to Ferdinand and Isabella of Spain?
Octagonal Kurotto
draw table with tikz matrix line shift a little bit
Left a meeting without apparent reason. What to do?
Why do zones differ for WGS84 UTM in Google Earth Pro vs. in QGIS 3.4?
What are the applications of the Mean Value Theorem?
Is it safe to run disc brakes up front on a mountain bike that lacks a disc caliper mounting point in the rear?
Should I pay closing cost and replace HVAC for buyer
Replacing triangulated categories with something better
How can I tell if I have simplified my talk too much?
Where do overtones in a 555 generated square wave come from?
opening Illustrator template file (.ait) for editing results in untitled.ai
Why don't my appliances work when my tester shows voltage at the outlets?
Asking an academic for a cursory look at a paper
Is it possible to build an escalator without iron, with 16th century technology?
Why is English not a regular language?
Why can't I book this multi-city fare on American Airlines?
Should I correct a mistake on an arXiv manuscript, that I found while refereeing it?
Googlebot crawls my page too often
How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?Googlebot craw rate is too slow for a huge siteHow to control old indexed page crawling rateGooglebot sends several hundred requests in a minuteGoogleBot crawling a page up to 3,000 times per dayGoogleBot is crawling my datepicker to inifnityHow often do search bots typically visit a website?Stop googlebot crawling URL more than once?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;
I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot).
But I have 1 article that have 300 page views in only 12 hours. I am curious, so I track the user agent and IP address and found that my page crawled by Googlebot. Too often.
My question is, what makes Google crawl this page too often, since other page is not? Do I need to worry about this?
googlebot crawl-rate
edited Oct 14 at 21:06
unor
19.6k3 gold badges33 silver badges92 bronze badges
asked Oct 14 at 3:15
RubahMalamRubahMalam
1381 silver badge6 bronze badges
add a comment
|
I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot).
But I have 1 article that have 300 page views in only 12 hours. I am curious, so I track the user agent and IP address and found that my page crawled by Googlebot. Too often.
My question is, what makes Google crawl this page too often, since other page is not? Do I need to worry about this?
googlebot crawl-rate
edited Oct 14 at 21:06
unor
19.6k3 gold badges33 silver badges92 bronze badges
asked Oct 14 at 3:15
RubahMalamRubahMalam
1381 silver badge6 bronze badges
2
Keep in mind, 300 over 12 hours is ~25 per hour, or approximately one every other minute. In terms of server load, that's practically nothing. If your page is huge, and that usage continues over an extended period of time, it could be a marginal problem for your bandwidth, but as others said it seems to be based on the URL's popularity, which means it should still remain a small fraction of the real traffic you get.
– Doktor J
Oct 14 at 19:25
There's a chance it might not actually be the real Googlebot. Here's how to find out support.google.com/webmasters/answer/80553?hl=en
– Tony McCreath
Oct 14 at 20:34
add a comment
|
I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot).
But I have 1 article that have 300 page views in only 12 hours. I am curious, so I track the user agent and IP address and found that my page crawled by Googlebot. Too often.
My question is, what makes Google crawl this page too often, since other page is not? Do I need to worry about this?
googlebot crawl-rate
edited Oct 14 at 21:06
unor
19.6k3 gold badges33 silver badges92 bronze badges
asked Oct 14 at 3:15
RubahMalamRubahMalam
1381 silver badge6 bronze badges
I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot).
But I have 1 article that have 300 page views in only 12 hours. I am curious, so I track the user agent and IP address and found that my page crawled by Googlebot. Too often.
My question is, what makes Google crawl this page too often, since other page is not? Do I need to worry about this?
googlebot crawl-rate
googlebot crawl-rate
edited Oct 14 at 21:06
unor
19.6k3 gold badges33 silver badges92 bronze badges
asked Oct 14 at 3:15
RubahMalamRubahMalam
1381 silver badge6 bronze badges
edited Oct 14 at 21:06
unor
19.6k3 gold badges33 silver badges92 bronze badges
asked Oct 14 at 3:15
RubahMalamRubahMalam
1381 silver badge6 bronze badges
edited Oct 14 at 21:06
unor
19.6k3 gold badges33 silver badges92 bronze badges
edited Oct 14 at 21:06
unor
19.6k3 gold badges33 silver badges92 bronze badges
edited Oct 14 at 21:06
unor
19.6k3 gold badges33 silver badges92 bronze badges
19.6k3 gold badges33 silver badges92 bronze badges
asked Oct 14 at 3:15
RubahMalamRubahMalam
1381 silver badge6 bronze badges
asked Oct 14 at 3:15
RubahMalamRubahMalam
1381 silver badge6 bronze badges
asked Oct 14 at 3:15
RubahMalamRubahMalam
1381 silver badge6 bronze badges
1381 silver badge6 bronze badges
2
Keep in mind, 300 over 12 hours is ~25 per hour, or approximately one every other minute. In terms of server load, that's practically nothing. If your page is huge, and that usage continues over an extended period of time, it could be a marginal problem for your bandwidth, but as others said it seems to be based on the URL's popularity, which means it should still remain a small fraction of the real traffic you get.
– Doktor J
Oct 14 at 19:25
There's a chance it might not actually be the real Googlebot. Here's how to find out support.google.com/webmasters/answer/80553?hl=en
– Tony McCreath
Oct 14 at 20:34
add a comment
|
2
Keep in mind, 300 over 12 hours is ~25 per hour, or approximately one every other minute. In terms of server load, that's practically nothing. If your page is huge, and that usage continues over an extended period of time, it could be a marginal problem for your bandwidth, but as others said it seems to be based on the URL's popularity, which means it should still remain a small fraction of the real traffic you get.
– Doktor J
Oct 14 at 19:25
There's a chance it might not actually be the real Googlebot. Here's how to find out support.google.com/webmasters/answer/80553?hl=en
– Tony McCreath
Oct 14 at 20:34
2
2
Keep in mind, 300 over 12 hours is ~25 per hour, or approximately one every other minute. In terms of server load, that's practically nothing. If your page is huge, and that usage continues over an extended period of time, it could be a marginal problem for your bandwidth, but as others said it seems to be based on the URL's popularity, which means it should still remain a small fraction of the real traffic you get.
– Doktor J
Oct 14 at 19:25
Keep in mind, 300 over 12 hours is ~25 per hour, or approximately one every other minute. In terms of server load, that's practically nothing. If your page is huge, and that usage continues over an extended period of time, it could be a marginal problem for your bandwidth, but as others said it seems to be based on the URL's popularity, which means it should still remain a small fraction of the real traffic you get.
– Doktor J
Oct 14 at 19:25
There's a chance it might not actually be the real Googlebot. Here's how to find out support.google.com/webmasters/answer/80553?hl=en
– Tony McCreath
Oct 14 at 20:34
There's a chance it might not actually be the real Googlebot. Here's how to find out support.google.com/webmasters/answer/80553?hl=en
– Tony McCreath
Oct 14 at 20:34
add a comment
|
2 Answers
2
active
oldest
votes
Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others.
URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.
Also, John Muller confirmed that on Twitter.
In addition, how Google's crawling works could also affect access to your website.
Some pages are known because Google has already crawled them before. Other pages are discovered when Google follows a link from a known page to a new page. Still other pages are discovered when a website owner submits a list of pages (a sitemap) for Google to crawl. If you're using a managed web host, such as Wix or Blogger, they might tell Google to crawl any updated or new pages that you make.
answered Oct 14 at 9:40
EmirodgarEmirodgar
3,2021 silver badge15 bronze badges
add a comment
|
There is no need to worry about this. In fact, it's a good thing, this way google would be able to index your updated content and would increase the chances of your page ranking higher.
If you, however, do not want to get your page indexed by google(for any reason whatsoever), there are multiple ways to do that.
You can add a noindex tag on the head of your page. Or if you want to try and reduce the crawl frequency, you can decrease the crawl rate
answered Oct 14 at 5:45
AnuveshAnuvesh
5091 gold badge1 silver badge15 bronze badges
add a comment
|
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "45"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fwebmasters.stackexchange.com%2fquestions%2f125681%2fgooglebot-crawls-my-page-too-often%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others.
URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.
Also, John Muller confirmed that on Twitter.
In addition, how Google's crawling works could also affect access to your website.
Some pages are known because Google has already crawled them before. Other pages are discovered when Google follows a link from a known page to a new page. Still other pages are discovered when a website owner submits a list of pages (a sitemap) for Google to crawl. If you're using a managed web host, such as Wix or Blogger, they might tell Google to crawl any updated or new pages that you make.
answered Oct 14 at 9:40
EmirodgarEmirodgar
3,2021 silver badge15 bronze badges
add a comment
|
Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others.
URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.
Also, John Muller confirmed that on Twitter.
In addition, how Google's crawling works could also affect access to your website.
Some pages are known because Google has already crawled them before. Other pages are discovered when Google follows a link from a known page to a new page. Still other pages are discovered when a website owner submits a list of pages (a sitemap) for Google to crawl. If you're using a managed web host, such as Wix or Blogger, they might tell Google to crawl any updated or new pages that you make.
answered Oct 14 at 9:40
EmirodgarEmirodgar
3,2021 silver badge15 bronze badges
add a comment
|
Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others.
URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.
Also, John Muller confirmed that on Twitter.
In addition, how Google's crawling works could also affect access to your website.
Some pages are known because Google has already crawled them before. Other pages are discovered when Google follows a link from a known page to a new page. Still other pages are discovered when a website owner submits a list of pages (a sitemap) for Google to crawl. If you're using a managed web host, such as Wix or Blogger, they might tell Google to crawl any updated or new pages that you make.
answered Oct 14 at 9:40
EmirodgarEmirodgar
3,2021 silver badge15 bronze badges
Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others.
URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.
Also, John Muller confirmed that on Twitter.
In addition, how Google's crawling works could also affect access to your website.
Some pages are known because Google has already crawled them before. Other pages are discovered when Google follows a link from a known page to a new page. Still other pages are discovered when a website owner submits a list of pages (a sitemap) for Google to crawl. If you're using a managed web host, such as Wix or Blogger, they might tell Google to crawl any updated or new pages that you make.
answered Oct 14 at 9:40
EmirodgarEmirodgar
3,2021 silver badge15 bronze badges
answered Oct 14 at 9:40
EmirodgarEmirodgar
3,2021 silver badge15 bronze badges
answered Oct 14 at 9:40
EmirodgarEmirodgar
3,2021 silver badge15 bronze badges
answered Oct 14 at 9:40
EmirodgarEmirodgar
3,2021 silver badge15 bronze badges
3,2021 silver badge15 bronze badges
add a comment
|
add a comment
|
There is no need to worry about this. In fact, it's a good thing, this way google would be able to index your updated content and would increase the chances of your page ranking higher.
If you, however, do not want to get your page indexed by google(for any reason whatsoever), there are multiple ways to do that.
You can add a noindex tag on the head of your page. Or if you want to try and reduce the crawl frequency, you can decrease the crawl rate
answered Oct 14 at 5:45
AnuveshAnuvesh
5091 gold badge1 silver badge15 bronze badges
add a comment
|
There is no need to worry about this. In fact, it's a good thing, this way google would be able to index your updated content and would increase the chances of your page ranking higher.
If you, however, do not want to get your page indexed by google(for any reason whatsoever), there are multiple ways to do that.
You can add a noindex tag on the head of your page. Or if you want to try and reduce the crawl frequency, you can decrease the crawl rate
answered Oct 14 at 5:45
AnuveshAnuvesh
5091 gold badge1 silver badge15 bronze badges
add a comment
|
There is no need to worry about this. In fact, it's a good thing, this way google would be able to index your updated content and would increase the chances of your page ranking higher.
If you, however, do not want to get your page indexed by google(for any reason whatsoever), there are multiple ways to do that.
You can add a noindex tag on the head of your page. Or if you want to try and reduce the crawl frequency, you can decrease the crawl rate
answered Oct 14 at 5:45
AnuveshAnuvesh
5091 gold badge1 silver badge15 bronze badges
There is no need to worry about this. In fact, it's a good thing, this way google would be able to index your updated content and would increase the chances of your page ranking higher.
If you, however, do not want to get your page indexed by google(for any reason whatsoever), there are multiple ways to do that.
You can add a noindex tag on the head of your page. Or if you want to try and reduce the crawl frequency, you can decrease the crawl rate
answered Oct 14 at 5:45
AnuveshAnuvesh
5091 gold badge1 silver badge15 bronze badges
answered Oct 14 at 5:45
AnuveshAnuvesh
5091 gold badge1 silver badge15 bronze badges
answered Oct 14 at 5:45
AnuveshAnuvesh
5091 gold badge1 silver badge15 bronze badges
answered Oct 14 at 5:45
AnuveshAnuvesh
5091 gold badge1 silver badge15 bronze badges
5091 gold badge1 silver badge15 bronze badges
add a comment
|
add a comment
|
Thanks for contributing an answer to Webmasters Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fwebmasters.stackexchange.com%2fquestions%2f125681%2fgooglebot-crawls-my-page-too-often%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
Keep in mind, 300 over 12 hours is ~25 per hour, or approximately one every other minute. In terms of server load, that's practically nothing. If your page is huge, and that usage continues over an extended period of time, it could be a marginal problem for your bandwidth, but as others said it seems to be based on the URL's popularity, which means it should still remain a small fraction of the real traffic you get.
– Doktor J
Oct 14 at 19:25
There's a chance it might not actually be the real Googlebot. Here's how to find out support.google.com/webmasters/answer/80553?hl=en
– Tony McCreath
Oct 14 at 20:34