#robots.txt file for http://www.communityguide.com.au User-agent: bingbot Crawl-Delay: 30 User-agent: NextGenSearchBot+1+(for+information+visit+http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx) Disallow:/ User-agent: NextGenSearchBot Disallow:/ User-agent: http://www.zoominfo.com Disallow:/ User-agent: NextGenSearchBot.aspx Disallow:/ User-agent: MegaIndex.ru/2.0 Disallow:/ User-agent: MegaIndex.ru/ Disallow:/ User-agent: ZoomInformation+Bot Disallow:/ User-agent: +https://www.megaindex.ru/ Disallow:/ User-agent: Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler) Disallow:/ User-agent: Mail.RU_Bot/2.0 Disallow:/ User-agent: Mail.RU_Bot/Robots/2.0 Disallow:/ User-agent: go.mail.ru/ Disallow:/ User-agent: +http://go.mail.ru/help/robots Disallow:/ User-agent: Mozilla/5.0+(compatible;+Linux+x86_64;+Mail.RU_Bot/Robots/2.0;++http://go.mail.ru/help/robots) Disallow:/ User-agent: Mozilla/5.0+(compatible;+Linux+x86_64;+Mail.RU_Bot/2.0;++http://go.mail.ru/help/robots) Disallow:/ User-agent: awooo Disallow:/ User-agent: compatible;Baiduspider/2.0;++http://www.baidu.com/search/spider.html Disallow:/ User-agent: Baiduspider/2.0 Disallow:/ User-agent: www.baidu.com/search/spider.html Disallow:/ User-agent: curl/7.29.0 Disallow:/ User-agent: curl/7.43.0 Disallow:/ User-agent: crawler/js-crawler Disallow:/ User-agent: js-crawler Disallow:/ User-agent: www.dataparksearch.org/bot Disallow:/ User-agent: DataparkSearch+(+http://www.dataparksearch.org/bot)+@+http://inet-sochi.ru/gday Disallow:/ User-agent: DataparkSearch Disallow:/ User-agent: EM7/ContentVerification/5.1+http://em7.sciencelogic.com Disallow:/ User-agent: ContentVerification/5.1 Disallow:/ User-agent: http://em7.sciencelogic.com Disallow:/ User-agent: ia_archiver Disallow:/ User-agent: InternetSeer.com Disallow:/ User-agent: Java/1.6.0_21 Disallow:/ User-agent: ltx71+-+(http://ltx71.com/) Disallow:/ User-agent: Java/1.4.1_04 Disallow:/ User-agent: Java/1.6.0_04 Disallow:/ User-agent: http://ltx71.com/ Disallow:/ User-agent: luis@cybo.com Disallow:/ User-agent: LWP::Simple/6.13+libwww-perl/6.13 Disallow:/ User-agent: MaxPointCrawler Disallow:/ User-agent: Nutch-1.10 Disallow:/ User-agent: maxpoint.crawler+at+maxpointinteractive+dot+com Disallow:/ User-agent: maxpoint.crawler Disallow:/ User-agent: maxpoint.crawler@maxpointinteractive.com Disallow:/ User-agent: Mozilla/5.0+(compatible;+AhrefsBot/5.1;++http://ahrefs.com/robot/) Disallow:/ User-agent: Mozilla/5.0+(compatible;+Cliqzbot/1.0++http://cliqz.com/company/cliqzbot) Disallow:/ User-agent: Cliqzbot/1.0 Disallow:/ User-agent: http://cliqz.com Disallow:/ User-agent: AhrefsBot/5.1 Disallow:/ User-agent: http://ahrefs.com/robot/ Disallow:/ User-agent: Mozilla/5.0+(compatible;+archive.org_bot;+Wayback+Machine+Live+Record;++http://archive.org/details/archive.org_bot) Disallow:/ User-agent: archive.org_bot Disallow:/ User-agent: http://archive.org/details/archive.org_bot Disallow:/ User-agent: Mozilla/5.0+(compatible;+archive.org_bot++http://www.archive.org/details/archive.org_bot) Disallow:/ User-agent: Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html) Disallow:/ User-agent: Baiduspider/2.0 Disallow:/ User-agent: http://www.baidu.com/search/spider.html Disallow:/ User-agent: www.baidu.com Disallow:/ User-agent: Mozilla/5.0+(compatible;+Dataprovider/6.92;++https://www.dataprovider.com/) Disallow:/ User-agent: Dataprovider/6.92 Disallow:/ User-agent: https://www.dataprovider.com/ Disallow:/ User-agent: Mozilla/5.0+(compatible;+DeuSu/5.0.2;++https://deusu.de/robot.html) Disallow:/ User-agent: DeuSu/5.0.2 Disallow:/ User-agent: https://deusu.de/robot.html Disallow:/ User-agent: Mozilla/5.0+(compatible;+DomainAppender+/1.0;++http://www.profound.net/domainappender) Disallow:/ User-agent: Mozilla/5.0+(compatible;+linkdexbot/2.2;++http://www.linkdex.com/bots/) Disallow:/ User-agent: linkdexbot/2.2 Disallow:/ User-agent: http://www.linkdex.com/bots/ Disallow:/ User-agent: DomainAppender+/1.0 Disallow:/ User-agent: http://www.profound.net/domainappender Disallow:/ User-agent: Mozilla/5.0+(compatible;+DotBot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com) Disallow:/ User-agent: DotBot/1.1 Disallow:/ User-agent: http://www.opensiteexplorer.org/dotbot Disallow:/ User-agent: Mozilla/5.0+(compatible;+DuckDuckGo-Favicons-Bot/1.0;++http://duckduckgo.com) Disallow:/ User-agent: DuckDuckGo Disallow:/ User-agent: http://duckduckgo.com Disallow:/ User-agent: Mozilla/5.0+(compatible;+Genieo/1.0+http://www.genieo.com/webfilter.html) Disallow:/ User-agent: Genieo/1.0 Disallow:/ User-agent: www.genieo.com/webfilter.html Disallow:/ User-agent: Mozilla/5.0+(compatible;+GrapeshotCrawler/2.0;++http://www.grapeshot.co.uk/crawler.php) Disallow:/ User-agent: GrapeshotCrawler/2.0 Disallow:/ User-agent: http://www.grapeshot.co.uk/crawler.php Disallow:/ User-agent: Mozilla/5.0+(compatible;+Linux+x86_64;+Mail.RU_Bot/2.0;++http://go.mail.ru/help/robots) Disallow:/ User-agent: Mail.RU_Bot/2.0 Disallow:/ User-agent: http://go.mail.ru/help/robots Disallow:/ User-agent: Mozilla/5.0+(compatible;+Linux+x86_64;+Mail.RU_Bot/Robots/2.0;++http://go.mail.ru/help/robots) Disallow:/ User-agent: Mail.RU_Bot/Robots/2.0 Disallow:/ User-agent: Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler) Disallow:/ User-agent: MegaIndex.ru/2.0 Disallow:/ User-agent: http://megaindex.com/crawler Disallow:/ User-agent: Mozilla/5.0+(compatible;+MojeekBot/0.6;++https://www.mojeek.com/bot.html) Disallow:/ User-agent: MojeekBot/0.6 Disallow:/ User-agent: https://www.mojeek.com/bot.html Disallow:/ User-agent: Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+LinkCheck+by+Siteimprove.com Disallow:/ User-agent: LinkCheck+by+Siteimprove.com Disallow:/ User-agent: Siteimprove.com Disallow:/ User-agent: Mozilla/5.0+(compatible;+MSIE+9.0;+Windows+NT+6.1;+Trident/5.0);+360Spider(compatible;+HaosouSpider;+http://www.haosou.com/help/help_3_2.html) Disallow:/ User-agent: 360Spider Disallow:/ User-agent: HaosouSpider Disallow:/ User-agent: http://www.haosou.com Disallow:/ User-agent: Mozilla/5.0+(compatible;+OrangeBot/2.0;+support.orangebot@orange.com) Disallow:/ User-agent: OrangeBot/2.0 Disallow:/ User-agent: support.orangebot@orange.com Disallow:/ User-agent: orange.com Disallow:/ User-agent: Mozilla/5.0+(compatible;+spbot/5.0.2;++http://OpenLinkProfiler.org/bot+) Disallow:/ User-agent: spbot/5.0.2 Disallow:/ User-agent: http://OpenLinkProfiler.org/bot Disallow:/ User-agent: Mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php) Disallow:/ User-agent: proximic Disallow:/ User-agent: www.proximic.com/info/spider.php Disallow:/ User-agent: www.proximic.com Disallow:/ User-agent: Mozilla/5.0+(compatible;+SurdotlyBot/1.0;++http://sur.ly/bot.html) Disallow:/ User-agent: SurdotlyBot/1.0 Disallow:/ User-agent: http://sur.ly/bot.htm Disallow:/ User-agent: Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots) Disallow:/ User-agent: YandexBot/3.0 Disallow:/ User-agent: http://yandex.com/bots Disallow:/ User-agent: Mozilla/5.0+(compatible;+yoozBot-2.2;+http://yooz.ir;+info@yooz.ir) Disallow:/ User-agent: yoozBot-2.2 Disallow:/ User-agent: http://yooz.ir;+info@yooz.ir Disallow:/ User-agent: yooz.ir Disallow:/ User-agent: Mozilla/5.0+(iPhone;+CPU+iPhone+OS+6_0+like+Mac+OS+X)+AppleWebKit/536.26+(KHTML,+like+Gecko)+++++++++++++++++Version/6.0+Mobile/10A5376e+Safari/8536.25+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot) Disallow:/ User-agent: SMTBot/1.0 Disallow:/ User-agent: http://www.similartech.com/smtbot Disallow:/ User-agent: Mozilla/5.0+(Windows+NT+6.1)+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot) Disallow:/ User-agent: Mozilla/5.0+(Windows+NT+6.1)+AppleWebKit/537.1+(KHTML,+like+Gecko)+Chrome/21.0.1180.89+Safari/537.1;+360Spider(compatible;+HaosouSpider;+http://www.haosou.com/help/help_3_2.html) Disallow:/ User-agent: NRLCorpusBuilder/Nutch-1.9 Disallow:/ User-agent: Nutch-1.9 Disallow:/ User-agent: nutch-1.4/Nutch-1.4 Disallow:/ User-agent: Nutch-1.4 Disallow:/ User-agent: Wotbox/2.01+(+http://www.wotbox.com/bot/) Disallow:/ User-agent: Wotbox/2.01 Disallow:/ User-agent: http://www.wotbox.com/bot/ Disallow:/ User-agent: SafeSearch+microdata+crawler+(https://safesearch.avira.com,+safesearch-abuse@avira.com) Disallow:/ User-agent: safesearch.avira.com Disallow:/ User-agent: SafeSearch+microdata+crawler Disallow:/ User-agent: Screaming+Frog+SEO+Spider/5.1 Disallow:/ User-agent: Spider/5.1 Disallow:/ User-agent: Screaming+Frog+SEO Disallow:/ User-agent: Slack-ImgProxy+1.136+(+https://api.slack.com/robots) Disallow:/ User-agent: Slack-ImgProxy+1.136 Disallow:/ User-agent: https://api.slack.com/robots Disallow:/ User-agent: Sogou+web+spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow:/ User-agent: Sogou+web+spider/4.0 Disallow:/ User-agent: www.sogou.com Disallow:/ User-agent: spiderman Disallow:/ User-agent: UXCrawlerBot Disallow:/ User-agent: Xenu+Link+Sleuth/1.3.8 Disallow:/ User-agent: Xenu+Link Disallow:/ User-agent: Sleuth/1.3.8 Disallow:/ User-agent: xSocks+v0.1 Disallow:/ User-agent: HETZNER-RZ-NBG-NET Disallow: / user-agent: AhrefsBot disallow: / User-Agent: DepSpid Disallow: / User-agent: ShopWiki Disallow: / User-Agent: MJ12bot Disallow: / User-agent: IRLbot Disallow: / User-agent: Twiceler Disallow: / User-agent: tutorgigbot Disallow: / User-Agent: OmniExplorer_Bot Disallow: / User-agent: e-SocietyRobot Disallow: / User-agent: turnitinbot Disallow: / User-Agent: sistrix Disallow: / User-agent: SurveyBot_IgnoreIP Disallow: / User-agent: archive.org_bot Disallow: / User-Agent: ia_archiver Disallow: / User-agent: spbot Disallow: / User-agent: BaiduMobaider Disallow: / User-agent: BaiduImagespider Disallow: / User-agent: Baiduspider Disallow: / User-agent: Sosospider Disallow: / User-agent: Semalt.com Disallow: / User-agent: yandex Disallow: / User-agent: Mozilla/5.0+(compatible;+Linux+x86_64;+Mail.RU_Bot/Img/2.0;++http://go.mail.ru/help/robots) Disallow: / User-agent: Mail.RU_Bot/Img/2.0 Disallow: / User-agent: http://go.mail.ru/help/robots Disallow: / User-agent: curl/7.19.7+(x86_64-unknown-linux-gnu)+libcurl/7.19.7+NSS/3.12.7.0+zlib/1.2.3+libidn/1.18+libssh2/1.2.2 Disallow: / User-agent: libssh2/1.2.2 Disallow: / User-agent: libidn/1.18 Disallow: / User-agent: zlib/1.2.3 Disallow: / User-agent: libcurl/7.19.7 Disallow: / User-agent: Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_1)+AppleWebKit/600.2.5+(KHTML,+like+Gecko)+Version/8.0.2+Safari/600.2.5+(Applebot/0.1;++http://www.apple.com/go/applebot) Disallow: / User-agent: Applebot/0.1 Disallow: / User-agent:www.apple.com/go/applebot Disallow: / User-agent: SafeDNSBot+(https://www.safedns.com/searchbot) Disallow: / User-agent: SafeDNSBot Disallow: / User-agent: safedns.com/searchbot Disallow: / User-agent: Mozilla/5.0+(compatible;+MJ12bot/v1.4.5;+http://www.majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.4.5 Disallow: / User-agent: majestic12.co.uk/bot.php Disallow: / # http://moz.com/products User-agent: dotbot Disallow: / # http://openlinkprofiler.org/bot User-agent: spbot Disallow: / # http://www.opensiteexplorer.org/dotbot # http://moz.com/researchtools/ose/dotbot User-agent: dotbot Disallow: / # https://megaindex.com/crawler User-agent: MegaIndex.ru Disallow: / User-agent: megaindex.com Disallow: / # http://www.moz.com/dp/rogerbot User-agent: rogerbot Disallow: / # http://http://www.semrush.com User-agent: SemrushBot Disallow: / User-agent: * Disallow: /cgadmin/ Disallow: /cgrsadmin/ Disallow: /siteadmin/ Disallow: /syteadmin/