Unsolved Replicate rogerbot error for server/hosting provider
-
Anyone got any ideas how to get a server/hosting provider who is preventing rogerbot from crawling and me not been able to set up a campaign to duplicate the error on there end?
The server/hosting provider is crazydomains dot com
My clients robots.txt
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
User-agent: rogerbotDisallow:
Sitemap: https://www. something
-
@tom-capper
Thanks Tom was more about server settings outside of robots.txt for hosting providers.
Managed to get them to remove rogerbot blocking on ALL there servers as they first stated that it was well known as a malicious robot hence the block???
Until I made them aware.... cough cough -
There are a bunch of third party tools that would allow you to validate robots.txt against a given user agent.
For example:
- https://technicalseo.com/tools/robots-txt/ (includes Moz user agents in dropdown)
- https://www.realrobotstxt.com/ (allows you to copy/paste a test robots file)
Without the hosting provider having a Moz account & campaign of their own, this is probably the easiest way to get them to replicate the error.
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What is the reason for the decline of my entire site in Google!
Why does my site rank so low in google results when the whole site was topping. It hit 200,000 impressions per day, and it's down to 1,000 impressions After this update 👇
Moz Tools | | A_h7med.-
Today we released the October 2022 Spam Update. Find out more about spam updates at This is why, and how do I know my site is spam This is a link to my site that you can see and check https://www.ktaabhulul.com/
B85CF67D-C76B-42A2-812A-9CCFE016D7FA.jpeg DF49D603-0D03-4A3B-926C-7C0136732804.jpeg0 -
Unsolved How to check the DA history?
Hello everyone!
Moz Tools | | ZeonAley
I want to know how to check the DA history of my website Inkjet Wholesale. Because I don't have any idea about Moz that how to use it. If anyone knows about DA's history check on Moz, please let me know.0 -
Unsolved Moz Crawl seems to be stuck?
Hi all, It seems like moz has been stuck on crawling our site for a while now - I had a message of 'you will get a notification for when your site crawl is complete' for about 2 weeks now, and it doesn't seem to finish it? Any ideas why this happens and how to fix it? Thank you in advance.
Moz Tools | | StevenWalley0 -
Unsolved Rogerbot blocked by cloudflare and not display full user agent string.
Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
Moz Pro | | BB_NPG
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
Ben Cloudflare activity log, showing differences in user agent strings
2022-07-01_13-05-59.png0 -
Unsolved Inaccurate backlinks being listed in Moz campaign
Moz keeps saying that these websites (amongst others)... https://toolszap.com/
Moz Tools | | JCN-SBWD
https://www.findwallpaperhd.com/
https://onlinecoursesschools.com/
https://www.tonzo.com/ ...are linking to one of my client sites at https://www.scamblermusic.com The websites in question clearly aren't related and aren't linking to the site, and there's no reference to Scambler Music anywhere in the source code. Why does Moz keep listing backlinks that simply don't appear to exist?0 -
Unsolved Anyone else finding that the Link Intersection tool is not accurate?
I ran the Link Intersect tool on my site vs competitor's site. The first 3 domains it tells me the competitor has links from but I do not are: medium.com, behance.net, and moz.com. I thought this was odd because I know for a fact I built profiles with a link on all 3 sites. So, I went to all 3 and found my links on these sites. I also double-checked in the Google search console and they show up there. Am I understanding the tool wrong? Is it really showing me sites we both have vs what I don't have?
Moz Tools | | pcannon0 -
Unsolved error in crawling
hello moz . my site is papion shopping but when i start to add it an error appears that it cant gather any data in moz!! what can i do>???
Moz Tools | | valigholami13860 -
Solved Help me to understannd Metadata Issues
I have 598 Title Pages that are "too long." I've attached an Excel to take a look. I think I'll need to go through it one by one to make corrections but do not know the 1, 2, 3 steps to take. If you can help after taking a look. Thanks! https://analytics.moz.com/pro/analytics/crawl/issues/group/meta/18696735/2887318?tab=title_too_long
Moz Tools | | BeckersBestShoes1