Top 165 Web Crawlers (Total 235) |
|
Name
|
Hits
|
1. |
Web Crawler - ia_archiver/1.4 (OSX 1.6; ) |
1 |
2. |
Web Crawler - ia_archiver/1.4 (Windows 2000 2.9; ) |
1 |
3. |
Web Crawler - ia_archiver/8.1 (Windows NT 5.6; ) |
1 |
4. |
Web Crawler - ia_archiver/1.7 (Windows 2000 7.8; ) |
1 |
5. |
Web Crawler - ia_archiver/4.4 (OSX 6.7; ) |
1 |
6. |
Web Crawler - ia_archiver/3.9 (OSX 4.7; ) |
1 |
7. |
Web Crawler - Yahoo! Slurp China |
1 |
8. |
Web Crawler - SheenBot/SheenBot-1.0.0 (Sheen web crawler) |
1 |
9. |
Web Crawler - Social Media Crawler using your Home URL on Twitter,Facebook,Myspace,Linkedin by ProfileCapture - contact profilecapture@gmail.com to report any problems with my crawling. http://profilecapture.com |
1 |
10. |
Web Crawler - Mozilla/2.0 (compatible; NEWT ActiveX; Win32) |
1 |
11. |
Web Crawler - ia_archiver/4.6 (Windows NT 2.1; ) |
1 |
12. |
Web Crawler - ia_archiver/4.5 (Windows NT 1.6; ) |
1 |
13. |
Web Crawler - ia_archiver/1.4 (Linux 7.7; ) |
1 |
14. |
Web Crawler - ia_archiver/5.1 (Windows NT 1.8; ) |
1 |
15. |
Web Crawler - ia_archiver/7.9 (Windows XP 1.1; ) |
1 |
16. |
Web Crawler - ia_archiver/1.8 (Linux 2.2; ) |
1 |
17. |
Web Crawler - ia_archiver/8.5 (Windows NT 7.2; ) |
1 |
18. |
Web Crawler - Yahoo! Slurp/Site Explorer |
1 |
19. |
Web Crawler - ia_archiver/7.3 (Windows 6.2; ) |
1 |
20. |
Web Crawler - ia_archiver/5.0 (OSX 6.4; ) |
1 |
21. |
Web Crawler - ia_archiver/3.7 (Windows XP 3.5; ) |
1 |
22. |
Web Crawler - Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) |
1 |
23. |
Web Crawler - CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com) |
1 |
24. |
Web Crawler - spotinfluence/Nutch-1.4 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com) |
1 |
25. |
Web Crawler - R6_FeedFetcher_(www.radian6.com/crawler) |
1 |
26. |
Web Crawler - Mozilla/5.0 (compatible; PathDefender/1.0; +http://www.pathdefender.com/help/crawler) |
1 |
27. |
Web Crawler - Googlebot/2.1 (+http://www.google.com/bot.html; For abuse: kilobyte2007@gmail.com) |
1 |
28. |
Web Crawler - REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project) |
1 |
29. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1 +http://www.googlebot.com/bot.html) |
1 |
30. |
Web Crawler - UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de) |
1 |
31. |
Web Crawler - Googlebot/2.1 |
1 |
32. |
Web Crawler - GarlikCrawler/1.1 (http://garlik.com/, crawler@garlik.com) |
1 |
33. |
Web Crawler - GeekTools Crawler - http://domains.geek-tools.org |
1 |
34. |
Web Crawler - ppffpbuumkxcfHakxcmdhont wrma x |
1 |
35. |
Web Crawler - rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org) |
1 |
36. |
Web Crawler - slurp, yahoo! slurp, slurp/2.0, inktomi slurp, slurp.so/1.0 |
1 |
37. |
Web Crawler - googlebot |
1 |
38. |
Web Crawler - FARSIGHTED60A_BU2_11B_HW (MRE/3.1.00(1280);MAUI/_6260A_V10a_IND_V04;BDATE/2013/06/21 23:48;LCD/240320;CHIP/MT6260;KEY/Normal;TOUCH/0;CAMERA/1;SENSOR/0;DEV/FARSIGHTED60A_BU2_11B_HW;WAP Browser/MAUI (HTTP PGDL;HTTPS);GMOBI/001;MBOUNCE/002;MOMA |
1 |
39. |
Web Crawler - Teemer (NetSeer, Inc.; http://www.netseer.com/crawler.html; crawler@netseer.com) |
1 |
40. |
Web Crawler - W3C_Validator/1.654 |
1 |
41. |
Web Crawler - ia_archiver/7.9 (Windows NT 6.3; ) |
1 |
42. |
Web Crawler - ia_archiver/3.0 (Windows XP 2.4; ) |
1 |
43. |
Web Crawler - ia_archiver/1.3 (Linux 3.6; ) |
1 |
44. |
Web Crawler - HRCrawlerBot |
1 |
45. |
Web Crawler - Googlebot/2.1 (http://www.google.bot.com/bot.html) |
1 |
46. |
Web Crawler - Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot) |
1 |
47. |
Web Crawler - TurnitinBot (https://turnitin.com/robot/crawlerinfo.html) |
1 |
48. |
Web Crawler - ia_archiver/4.6 (OSX 6.2; ) |
1 |
49. |
Web Crawler - ia_archiver/3.2 (Windows NT 3.9; ) |
1 |
50. |
Web Crawler - ia_archiver/8.2 (Windows NT 4.4; ) |
1 |
51. |
Web Crawler - Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com) |
1 |
52. |
Web Crawler - Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com) |
1 |
53. |
Web Crawler - ia_archiver/5.3 (Linux 1.9; ) |
1 |
54. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/blog/last.php) |
1 |
55. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/last.php) |
1 |
56. |
Web Crawler - VelenPublicWebCrawler/0.1 |
1 |
57. |
Web Crawler - Contacts-Crawler/0.2 (+http://www.scrapinghub.com) |
1 |
58. |
Web Crawler - WebZIP/x.x (http://www.spidersoft.com) |
1 |
59. |
Web Crawler - IP-Guide.com Crawler/1.0 (https://ip-guide.com) |
1 |
60. |
Web Crawler - Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler) |
1 |
61. |
Web Crawler - msnbot/1.0 ( http://search.msn.com/msnbot.htm) |
1 |
62. |
Web Crawler - FriendlyCrawler/Nutch-1.20-SNAPSHOT |
2 |
63. |
Web Crawler - Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; ) |
2 |
64. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/vb/index.php) |
2 |
65. |
Web Crawler - Willow Internet Crawler by Twotrees V2.1 |
2 |
66. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://abwc.iespana.es/last.php) |
2 |
67. |
Web Crawler - DotBot/1.0.1 (http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org) |
2 |
68. |
Web Crawler - Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru) |
2 |
69. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://abwc.iespana.es/forum/index.php) |
2 |
70. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) |
2 |
71. |
Web Crawler - RMA/1.0 (compatible; RealMedia) |
2 |
72. |
Web Crawler - ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html) |
2 |
73. |
Web Crawler - YPBot/Raven1.1.3 (compatible; Googlebot/2.1;+http://www.yellowpages.com/about/legal/crawl) |
2 |
74. |
Web Crawler - ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) |
2 |
75. |
Web Crawler - SheenBot/SheenBot-1.0.4 (Sheen web crawler) |
2 |
76. |
Web Crawler - KSCrawler/Nutch-1.0 (http://www.kindsight.net/en/kscrawler; crawler@kindsight.net) |
2 |
77. |
Web Crawler - ZoomInformation Bot |
2 |
78. |
Web Crawler - Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions. |
2 |
79. |
Web Crawler - Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
2 |
80. |
Web Crawler - HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko) |
2 |
81. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler) |
2 |
82. |
Web Crawler - LivelapBot/0.2 (http://site.livelap.com/crawler) |
2 |
83. |
Web Crawler - ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company.; http://www.ilial.com/crawler; crawl@ilial.com) |
2 |
84. |
Web Crawler - Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com) |
2 |
85. |
Web Crawler - Utopia WebWasher 3.0 |
2 |
86. |
Web Crawler - rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)/Nutch-1.13 |
2 |
87. |
Web Crawler - ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) |
2 |
88. |
Web Crawler - Jayde crawler |
3 |
89. |
Web Crawler - UCMore Crawler App |
3 |
90. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html) |
3 |
91. |
Web Crawler - CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp) |
3 |
92. |
Web Crawler - Mozilla/5.0 (compatible; WebHistoryBot/1.2.1 IS NOT SE bot like Googlebot/2.1; +http://www.google.com/bot.html,Yahoo! Slurp or Bingbot) |
3 |
93. |
Web Crawler - Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler; https://github.com/DigitalPebble/storm-crawler; stormcrawler@digitalpebble.com) |
3 |
94. |
Web Crawler - ahvemnkucouiqrmntldg9ylrmauqxl |
3 |
95. |
Web Crawler - Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/) |
3 |
96. |
Web Crawler - W3C_Validator/1.3 http://validator.w3.org/services |
3 |
97. |
Web Crawler - LinksCrawler 0.1beta |
3 |
98. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html |
3 |
99. |
Web Crawler - Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com) |
3 |
100. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com) |
3 |
101. |
Web Crawler - ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) |
3 |
102. |
Web Crawler - HouxouCrawler/Nutch-0.9 (houxou.com's nutch-based crawler which serves special interest on-line communities; http://www.houxou.com/crawler; crawler at houxou dot com) |
3 |
103. |
Web Crawler - SheenBot/SheenBot-1.0.3 (Sheen web crawler) |
3 |
104. |
Web Crawler - Jayde Crawler. http://www.jayde.com |
3 |
105. |
Web Crawler - DotBot/1.0.1 (http://www.dotnetdotcom.org/#info, crawler@dotnetdotcom.org) |
3 |
106. |
Web Crawler - Mozilla/5.0 (compatible; NOT Googlebot/2.1; +http://www.google.com/bot.html) |
4 |
107. |
Web Crawler - SheenBot/SheenBot-1.0.1 (Sheen web crawler) |
4 |
108. |
Web Crawler - UXCrawlerBot |
4 |
109. |
Web Crawler - SheenBot/SheenBot-1.0.2 (Sheen web crawler) |
4 |
110. |
Web Crawler - abnmwnanoifid5hkxn hdmwrmaDDirls5ngpnk |
4 |
111. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler) |
4 |
112. |
Web Crawler - DomainCrawler/1.0 |
4 |
113. |
Web Crawler - Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (for mobile crawler) ) |
4 |
114. |
Web Crawler - Attributor/Nutch-1.0-dev (Test crawler; http://www.attributor.com; info at attributor com) |
4 |
115. |
Web Crawler - Quick-Crawler (+https://www.scrapinghub.com/) |
4 |
116. |
Web Crawler - Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io) |
4 |
117. |
Web Crawler - CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) |
5 |
118. |
Web Crawler - SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) |
5 |
119. |
Web Crawler - Python-urllib/2.7 (+http://example.com/crawler.html) |
5 |
120. |
Web Crawler - http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html |
5 |
121. |
Web Crawler - Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/) |
5 |
122. |
Web Crawler - Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620 |
5 |
123. |
Web Crawler - ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp) |
5 |
124. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/forum/index.php) |
6 |
125. |
Web Crawler - blogtop.us crawler - http://blogtop.us/ |
6 |
126. |
Web Crawler - ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com) |
6 |
127. |
Web Crawler - yyqnnlrcewrmaitmsimi |
6 |
128. |
Web Crawler - Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com) |
6 |
129. |
Web Crawler - Zauba Crawler/1.0 (Zauba Search for Research; http://www.zauba.io/; admin@zauba.io) |
6 |
130. |
Web Crawler - Mozilla/5.0 (compatible; heritrix/3.0.0 +http://dataclip.com/crawler.html) |
7 |
131. |
Web Crawler - Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0 |
7 |
132. |
Web Crawler - Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0 |
8 |
133. |
Web Crawler - Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru) |
8 |
134. |
Web Crawler - Comodo-Webinspector-Crawler 2.1 |
8 |
135. |
Web Crawler - WebCrawler |
8 |
136. |
Web Crawler - rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com) |
8 |
137. |
Web Crawler - Mozilla/5.0 (compatible; TestCrawler) |
9 |
138. |
Web Crawler - msnbot/2.0b |
9 |
139. |
Web Crawler - SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info at similarpages dot com) |
9 |
140. |
Web Crawler - A .NET Web Crawler |
9 |
141. |
Web Crawler - DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/free-islam.com |
9 |
142. |
Web Crawler - Friendly_Crawler/Nutch-1.20-SNAPSHOT |
10 |
143. |
Web Crawler - DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/free-islam.net) |
10 |
144. |
Web Crawler - DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/free-islam.com) |
10 |
145. |
Web Crawler - DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/free-islam.org) |
11 |
146. |
Web Crawler - DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/free-islam.net |
11 |
147. |
Web Crawler - GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com) |
11 |
148. |
Web Crawler - CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com |
14 |
149. |
Web Crawler - GoogleBot 1.0 |
15 |
150. |
Web Crawler - Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions. |
15 |
151. |
Web Crawler - AdnormCrawler www.adnorm.com/crawler |
16 |
152. |
Web Crawler - Googlebot (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
16 |
153. |
Web Crawler - Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler) |
17 |
154. |
Web Crawler - Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/) |
17 |
155. |
Web Crawler - MSR-ISRCCrawler |
18 |
156. |
Web Crawler - CRAZYWEBCRAWLER 0.9.9, http://www.crazywebcrawler.com |
18 |
157. |
Web Crawler - CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com |
19 |
158. |
Web Crawler - VelenPublicWebCrawler (velen.io) |
19 |
159. |
Web Crawler - www.integromedb.org/Crawler |
20 |
160. |
Web Crawler - CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com |
20 |
161. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) |
21 |
162. |
Web Crawler - ConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm) |
23 |
163. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html) |
25 |
164. |
Web Crawler - BacklinkCrawler (http://www.backlinktest.com/crawler.html) |
25 |
165. |
Web Crawler - GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com) |
25 |