Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Mozscape API
    • Free SEO Tools
      • Competitive Research
      • Link Explorer
      • Keyword Explorer
      • Domain Analysis
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your All-In-One Suite of SEO Tools

        The essential SEO toolset: keyword research, link building, site audits, page optimization, rank tracking, reporting, and more.

        Learn more
        Try Moz Pro free
        Illustration of Moz Pro
      • Moz Local

        Complete Local SEO Management

        Raise your local SEO visibility with easy directory distribution, review management, listing updates, and more.

        Learn more
        Check my presence
        Illustration of Moz Local
      • STAT

        Enterprise Rank Tracking

        SERP tracking and analytics for SEO experts, STAT helps you stay competitive and agile with fresh insights.

        Learn more
        Book a demo
        Illustration of STAT
      • Mozscape API

        The Power of Moz Data via API

        Power your SEO with the proven, most accurate link metrics in the industry, powered by our index of trillions of links.

        Learn more
        Get connected
        Illustration of Mozscape API
      • Compare SEO Products
    • Free SEO Tools
      • Competitive Research

        Competitive Intelligence to Fuel Your SEO Strategy

        Gain intel on your top SERP competitors, keyword gaps, and content opportunities.

        Find competitors
        Illustration of Competitive Research
      • Link Explorer

        Powerful Backlink Data for SEO

        Explore our index of over 40 trillion links to find backlinks, anchor text, Domain Authority, spam score, and more.

        Get link data
        Illustration of Link Explorer
      • Keyword Explorer

        The One Keyword Research Tool for SEO Success

        Discover the best traffic-driving keywords for your site from our index of over 500 million real keywords.

        Search keywords
        Illustration of Keyword Explorer
      • Domain Analysis

        Free Domain SEO Analysis Tool

        Get top competitive SEO metrics like Domain Authority, top pages, ranking keywords, and more.

        Analyze domain
        Illustration of Domain Analysis
      • MozBar

        Free, Instant SEO Metrics As You Surf

        Using Google Chrome, see top SEO metrics instantly for any website or search result as you browse the web.

        Try MozBar
        Illustration of MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

        Read the Beginner's Guide
      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

        See All SEO Guides
      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

        Visit the Learning Center
      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

        Explore the Catalog
      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

        View All Webinars
      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

        Find SEO Answers
      The Impact of Local Business Reviews
      SEO Industry Report

      The Impact of Local Business Reviews

      Learn more
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

        Grow Your Business
      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

        Read Our Story
      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

        Drive Client Success
      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

        See What's Possible
      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

        Scale Your SEO
      • New Releases

        Get the scoop on the latest and greatest from Moz.

        See What’s New
      Surface actionable competitive intel
      New Feature: Moz Pro

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Mozscape API
      • Mozscape API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. Moz Tools
    3. Getting Started
    4. Our crawler was not able to access the robots.txt file on your site

    Our crawler was not able to access the robots.txt file on your site

    Getting Started
    4
    5
    1800
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • tigersohelll
      tigersohelll last edited by

      Hello Mozzers!

      I've received an error message saying the site can't be crawled because Moz is unable to access the robots.txt. I've spoken to the webmaster and he can't understand why the robot.txt can't be accessed in Moz.

      https://www.thefurnshop.co.uk/robots.txt

      and Google isn't flagging anything up to us.

      Does anyone know how to solve this problem?

      Thanks

      1 Reply Last reply Reply Quote 0
      • jaytarr
        jaytarr Subscriber @LoganRay last edited by

        @LoganRay This was our issue. Didn't know Moz tries to retrieve the HTTP robots.txt first. Our HTTPS redirect was not working on static files only, so the HTTP path to the robots.txt was failing. We did not notice it because the HSTS policy was forcing the browser to redirect.

        1 Reply Last reply Reply Quote 0
        • LoganRay
          LoganRay @LoganRay last edited by

          Wanted to jump back in on this topic as I've just confirmed my initial suspicion.

          I just added a new client to our Moz account and had the exact same issue, crawler unable to access the robots.txt file. It's a secure site and was configured in Moz without the HTTPS. When I go to the robots.txt file without https://www, it redirects to the same thing as yours where the / between the TLD and page path gets removed.

          Reconfigure your site and it should begin to work.

          jaytarr 1 Reply Last reply Reply Quote 3
          • Tenlo
            Tenlo last edited by

            There are 2 parts of your robots.txt that could be causing this, and it all just depends on how each bot is reading regular expressions in your robots.txt:

            First, your Disallow: /? can be read as Disallow all paths starting with "/" with 0 to infinity characters "" and one character "?". Try replacing this part with Disallow: /*? to make it not crawl anything with a query string (which is what I believe you were going for).

            Second, you have a open Disallow followed by the User-agent: rogerbot and while this should not be read this way, once again it all depends on how each bot reads the commands. To fix this you should change your Disallow following your Googlebot-Image as Disallow: /

            1 Reply Last reply Reply Quote 0
            • LoganRay
              LoganRay last edited by

              Hi there,

              There's something odd going on when I try to access your robots.txt file without the www. The www gets added back on, but when it does, the slash between the TLD and page path gets deleted, see below. I'm guessing your domain in Moz is configured without the www, which means RogerBot is getting redirected to this slash-less version of the file.

              https://www.thefurnshop.co.ukrobots.txt

              LoganRay 1 Reply Last reply Reply Quote 2
              • 1 / 1
              • First post
                Last post

              Got a burning SEO question?

              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


              Start my free trial


              Browse Questions

              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

              • SEO Tactics

                Discuss the SEO process with fellow marketers

              • Community

                Discuss industry events, jobs, and news!

              • Digital Marketing

                Chat about tactics outside of SEO

              • Research & Trends

                Dive into research and trends in the search industry.

              • Support

                Connect on product support and feature requests.

              • See all categories

              Related Questions

              • integratedproperty

                What is the quickest and easiest way to run an SEO audit on a Wordpress site that at least shows all the mechanical problems?

                What is the quickest and easiest way to run an SEO audit on a Wordpress site that at least shows all the mechanical problems?

                Getting Started | | integratedproperty
                0
              • markwillson

                How I can increase DA of my site?

                Hi, I have my  4 month old blog, PA of site is 17 but DA is still 5. I don't know how to increase DA of site.  Please suggest me how to increase DA of Site https://myeasygrader.com/  . Thanks

                Getting Started | | markwillson
                0
              • BloggyMoms

                Can't Crawl Site - but deducting crawls.

                Why am I being deducted crawls if MOZ keeps telling me that it can't crawl my site?

                Getting Started | | BloggyMoms
                1
              • Webreviewadmin

                When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?

                Hi guys, When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file. However, my site can be crawled by SEM Rush with no mention of problems with roots.txt file issues. My developer has looked into it and insists their is no problem with my robots.txt and I've tried the Moz crawl at least 6 times over an 8 week period. Has anyone ever seen such a large discrepancy between Moz and SEM Rush or have any ideas why Moz has this issue with my site?? TIA everyone

                Getting Started | | Webreviewadmin
                0
              • monkeex

                Moz Site Crawl can't index WIX sites

                We've been attempting to work on some SEO for a new potential client however they are using a WIX site. We've noticed that Moz SEO tools will not index any WIX sites. e.g. https://www.sharonradisch.com/ (which is one of their case studies). Anyone seen this that can offer any advice? Thanks,
                Mark

                Getting Started | | monkeex
                2
              • btreloar

                Standard Syntax in robots.txt doesn't prevent Moz bot from crawling

                A client is getting many false positive site crawl errors for things like duplicate titles and duplicate content on pages that include /tag/ in the URL. An example is https://needquest.com/place_tag/autism-spectrum-disorder/page/4/ To resolve this we have set up a disallow statement in the robots.txt file that says
                Disallow: /page/ For some reason this appears not to work, as the site crawl errors continue to list pages like this. Does anyone understand why that would be and what we need to do to properly disallow crawling these pages?

                Getting Started | | btreloar
                0
              • IV-Debbie

                In Open site explorer the page title and Url show in the left hand column. Why do some of my pages have no data for page title?

                I am a first time user.  Newly updated site using Drupal and having lots of SEO problems.  Under site explorer, several pages list NO DATA for the page title.  This doesn't seem right.  Any suggestions on what this means?

                Getting Started | | IV-Debbie
                0
              • trumpfinc

                Site traffic vs other sites

                I have a quick question. Does Moz have a tool I can use to see how much traffic my site gets a month versus some competition? I know Alexa has this but you have to create an account. I am just wondering if Moz has one and I am missing it.

                Getting Started | | trumpfinc
                0
              Moz logo
              • Contact
              • Community
              • Free Trial
              • Terms & Privacy
              • Jobs
              • Help
              • News & Press
              • Mozcon
              © 2021 - 2023 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.

              Looks like your connection to Moz was lost, please wait while we try to reconnect.