Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Mozscape API
    • Free SEO Tools
      • Competitive Research
      • Link Explorer
      • Keyword Explorer
      • Domain Analysis
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your All-In-One Suite of SEO Tools

        The essential SEO toolset: keyword research, link building, site audits, page optimization, rank tracking, reporting, and more.

        Learn more
        Try Moz Pro free
        Illustration of Moz Pro
      • Moz Local

        Complete Local SEO Management

        Raise your local SEO visibility with easy directory distribution, review management, listing updates, and more.

        Learn more
        Check my presence
        Illustration of Moz Local
      • STAT

        Enterprise Rank Tracking

        SERP tracking and analytics for SEO experts, STAT helps you stay competitive and agile with fresh insights.

        Learn more
        Book a demo
        Illustration of STAT
      • Mozscape API

        The Power of Moz Data via API

        Power your SEO with the proven, most accurate link metrics in the industry, powered by our index of trillions of links.

        Learn more
        Get connected
        Illustration of Mozscape API
      • Compare SEO Products
    • Free SEO Tools
      • Competitive Research

        Competitive Intelligence to Fuel Your SEO Strategy

        Gain intel on your top SERP competitors, keyword gaps, and content opportunities.

        Find competitors
        Illustration of Competitive Research
      • Link Explorer

        Powerful Backlink Data for SEO

        Explore our index of over 40 trillion links to find backlinks, anchor text, Domain Authority, spam score, and more.

        Get link data
        Illustration of Link Explorer
      • Keyword Explorer

        The One Keyword Research Tool for SEO Success

        Discover the best traffic-driving keywords for your site from our index of over 500 million real keywords.

        Search keywords
        Illustration of Keyword Explorer
      • Domain Analysis

        Free Domain SEO Analysis Tool

        Get top competitive SEO metrics like Domain Authority, top pages, ranking keywords, and more.

        Analyze domain
        Illustration of Domain Analysis
      • MozBar

        Free, Instant SEO Metrics As You Surf

        Using Google Chrome, see top SEO metrics instantly for any website or search result as you browse the web.

        Try MozBar
        Illustration of MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

        Read the Beginner's Guide
      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

        See All SEO Guides
      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

        Visit the Learning Center
      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

        Explore the Catalog
      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

        View All Webinars
      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

        Find SEO Answers
      The Impact of Local Business Reviews
      SEO Industry Report

      The Impact of Local Business Reviews

      Learn more
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

        Grow Your Business
      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

        Read Our Story
      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

        Drive Client Success
      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

        See What's Possible
      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

        Scale Your SEO
      • New Releases

        Get the scoop on the latest and greatest from Moz.

        See What’s New
      Surface actionable competitive intel
      New Feature: Moz Pro

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Mozscape API
      • Mozscape API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. What is a good crawl budget?

    What is a good crawl budget?

    Technical SEO
    4
    4
    2396
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • yaelslater
      yaelslater Subscriber last edited by

      Hi Community!

      I am in the process of updating sitemaps and am trying to obtain a standard for what is considered "strong" crawl budget? Every documentation I've found includes how to make it better or what to watch out for. However, I'm looking for an amount to obtain for (ex: 60% of the sitemap has been crawled, 100%, etc.)

      BlueprintMarketing 1 Reply Last reply Reply Quote 1
      • mywebguytaylor
        mywebguytaylor @BlueprintMarketing last edited by

        @blueprintmarketing I have a large website with Wordpress image folders going back to 2009.

        I am currently redesigning my website, and I am trying to determine if there is any benefit to trying to shrink down / delete those images and image folders which I am no longer using.

        I really do not have time to go through all of those image folders, and see which ones I am still using, and which ones I am not using anymore. I am hoping this does not matter?

        Does anyone here know if this matters when it comes to Google's Crawl Budget?

        All of the images are completely optimized and crunched, however, my question is whether it would be worth the time investment to go through every single folder and thousands of images and try to delete the ones which are not being referenced on any of my pages?

        Does anyone have a definitive answer regarding Crawl Budget?

        1 Reply Last reply Reply Quote 0
        • seopack.or3best
          seopack.or3best Subscriber @BlueprintMarketing last edited by

          @blueprintmarketing

          Can you give some inputs about the site [https://indiapincodes.net/](link url) I tried all recommendations, only 30% of the url is been indexed. would appreciate your time.

          1 Reply Last reply Reply Quote 0
          • BlueprintMarketing
            BlueprintMarketing @yaelslater last edited by BlueprintMarketing

            @yaelslater
            Unless you have a huge site, I'm talking about half a million to one million pages. I Would not worry about True Google crawl budge anymore.

            However, if only 60% of URLs in your XML site map are being indexed, make sure they are indexable URLs if they're not index value, or else you should be able to click in the coverage section of Search Console. It will give you a reason why your URL was submitted by an XML site map or not noindex.
            A recent study showed about 20% of URLs on all websites across the study were not indexed for one reason or another.

            But make sure there are only 200 URLs, no redirects 301, 302, or 404's or noindex nofollow URLs in the XML sitemap because obviously, Google does not put them into the index if the Search Console does not tell you the issue & you would like to share your domain with me, I'm sure I could figure it out.

            I don't know if you're using a CDN and if you could share a little more with me especially the domain I can be a lot more helpful.

            You could also use a tool like screaming frog and generate a new site map and make sure that is not the issue. If you're using Yoast, you can turn it on and off if you wanted to create a new site map.

            You can create up to 500 pages for free using Screaming Frog SEO Spider it is paid after that https://www.screamingfrog.co.uk/xml-sitemap-generator/

            Or if you want it or you can generate over 1000 URLs for free online I would recommend https://www.sureoak.com/seo-tools/google-xml-sitemap-generator
            However, please keep in mind the sureoak tool has things like a "keyword density checker" that makes me feel like this site is giving out that information because that's not a real thing that Google considers unless you use the same word for every word in the document. Keyword density is one of those things that are not real
            But the XML site map generator works just fine

            I hope this was of help,
            Tom

            seopack.or3best mywebguytaylor 2 Replies Last reply Reply Quote 2
            • 1 / 1
            • First post
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Browse Questions

            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

            • SEO Tactics

              Discuss the SEO process with fellow marketers

            • Community

              Discuss industry events, jobs, and news!

            • Digital Marketing

              Chat about tactics outside of SEO

            • Research & Trends

              Dive into research and trends in the search industry.

            • Support

              Connect on product support and feature requests.

            • See all categories

            Related Questions

            • abisti2

              Increase in Crawl Errors

              I had a problem with a lot of crawl errors (on Google Search Console) a while back, due to the removal of a shopping cart. I thought I'd dealt with this & Google seemed to agree (see attached pic), but now they're all back with a vengeance! The crawl errors are all the old shop pages that I thought I'd made clear weren't there anymore. The sitemaps (using Yoast on Wordpress to generate these) all updated 16 Aug but the increase didn't happen till 18-20. How do I make it clear to Google that these pages are gone forever? Screen-Shot-2016-08-22-at-10.19.05.png

              Technical SEO | | abisti2
              0
            • Bill_K

              Keyword rankings for new website good in Yahoo and Bing but no movement in Google?

              Hello, This past September I launched a new redesigned website for a client. His old website was a static html site that was many years old and the new website was created using WordPress. With the new design we made sure to use all the proper techniques for SEO. (h1 tages, image names, quality links, page titles, etc.) Plus, all the content is new content written for this site. I've actually launched new sites many times and after a few months usually start seeing keyword ranking improvements from the major search engines. With this particular website I'm seeing improvements in Yahoo and Bing but no movement in Google. I've used Google Webmaster Tools and made sure my sitemap is being submitted, etc. It all seems good, but I can't understand why Yahoo and Bing are working but nothing from Google. My page grades are all A's and B's and Moz isn't showing any big issues. Maybe I need to give it more time? This client is a lawyer and has many websites out there so maybe he's being penalized somewhere I don't know? As I mentioned, I've been doing SEO for about 8 years and have never had this much trouble with Google. I was wondering if you can look at the site and see if there are any glaring issues I'm missing. Website: http://www.arizonamedicalmalpractice.info/ The keyword phrases were are looking at are "Phoenix Medical Malpractice Lawyer", "Phoenix Medical Malpractice Attorney:, "Arizona Medical Malpractice Lawyer & Attorney", etc. I appreciate anyone who takes the time and does a quick look over. Thanks very much, Bill

              Technical SEO | | Bill_K
              0
            • Web-Incite

              Pages appear fine in browser but 404 error when crawled?

              I am working on an eCommerce website that has been written in WordPress with the shop pages in E commerce Plus PHP v6.2.7. All the shop product pages appear to work fine in a browser but 404 errors are returned when the pages are crawled. WMT also returns a 404 error when ‘fetch as Google’  is used. Here is a typical page: http://www.flyingjacket.com/proddetail.php?prod=Hepburn-Jacket Why is this page returning a 404 error when crawled? Please help?

              Technical SEO | | Web-Incite
              0
            • TheSEOGuy1

              Templates for Meta Description, Good or Bad?

              Hello, We have a website where users can browse photos of different categories. For each photo we are using a meta description template such as: Are you looking for a nice and cool photo? [Photo name] is the photo which might be of interest to you. And in the keywords tags we are using: [Photo name] photos, [Photo name] free photos, [Photo name] best photos. I'm wondering, is this any safe method? it's very difficult to write a manual description when you have 3,000+ photos in the database. Thanks!

              Technical SEO | | TheSEOGuy1
              0
            • ClaireH-184886

              Can anyone show me a good example of Microdata for a site

              Hi, i am just learning about microdata after being told that it would be a good idea to have this for my site www.in2town.co.uk  as i understand it, if i put this in, then it will allow google to understand more about my page I have been reading about it here https://support.google.com/webmasters/bin/answer.py?hl=en&answer=99170 and would like to know if anyone is using it and if it is making a difference, also can anyone show me some good examples on a working website please i work in joomla so as i understand it, i would have to put this in each page to help google understand more about the page and its content any help would be great.

              Technical SEO | | ClaireH-184886
              0
            • beck398

              Pagination/Crawl Errors

              Hi, Ive only just joined SEO moz and after they crawled my site they came up with 3600  crawl errors mostly being duplicate content and duplicate urls. After researching  this it soon became clear it was due to on page pagination and after speaking with Abe from SEO mozhe advised me to take action by getting our developers to implement rel=”next” & rel=”prev” to review. soon after our developers implemented this code ( I have no understanding of this what so ever) 90% of my keywords I had been ranking for in the top 10 have dropped out the top 50! Can anyone explain this or help me with this? Thanks Andy

              Technical SEO | | beck398
              0
            • Linesides

              SEOMoz Crawl Diagnostic indicates duplicate page content for home page?

              My first SEOMoz Crawl Diagnostic report for my website indicates duplicate page content for my home page. It lists the home page URL Page Title and URL twice. How do I go about diagnosing this? Is the problem related to the following code that is in my .htaccess file? (The purpose of the code was to redirect any non "www" backlink referrals to the "www" version of the domain.) RewriteCond %{HTTP_HOST} ^whatever.com [NC]
              RewriteRule ^(.*)$ http://www.whatever.com/$1 [L,R=301] Should I get rid of the "http" reference in the second line? Related to this is a notice in the "Crawl Notices Found" -- "301 Permanent redirect" which shows my home page title as "http://whatever.com" and shows the redirect address as http://http://www.whatever.com/ I'm guessing this problem is again related to the redirect code I'm using. Also... The report indicates duplicate content for those links that have different parameters added to the URL i.e. http://www.whatever.com?marker=Blah Blah&markerzoom=13 If I set up a canonical reference for the page, will this fix this? Thank you.

              Technical SEO | | Linesides
              0
            • soulnafein

              Google crawl rate almost zero since re-launch, organic search up 50% though!

              We're confused as to why Google's crawl of our site has dropped hugely since our new site went live. The URLs of almost all pages changed, and were 301d to the new site. About 20% of our pages were blocked by robots.txt for the re-launch. The re-launch has been great for organic search, with hits up about 50%. Yet our new content is taking a lot longer to get indexed than before. Our KB downloaded a day according to webmaster tools are well down, as is time spent downloading a page. Any ideas as to why this is?i7hwX.png

              Technical SEO | | soulnafein
              0
            Moz logo
            • Contact
            • Community
            • Free Trial
            • Terms & Privacy
            • Jobs
            • Help
            • News & Press
            • Mozcon
            © 2021 - 2023 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.

            Looks like your connection to Moz was lost, please wait while we try to reconnect.