Top 165 Web Crawlers (Total 235) |
|
Name
|
Hits
|
1. |
Web Crawler - ZoomInformation Bot |
2 |
2. |
Web Crawler - Zauba Crawler/1.0 (Zauba Search for Research; http://www.zauba.io/; admin@zauba.io) |
6 |
3. |
Web Crawler - yyqnnlrcewrmaitmsimi |
6 |
4. |
Web Crawler - YPBot/Raven1.1.3 (compatible; Googlebot/2.1;+http://www.yellowpages.com/about/legal/crawl) |
2 |
5. |
Web Crawler - Yahoo! Slurp/Site Explorer |
1 |
6. |
Web Crawler - Yahoo! Slurp China |
1 |
7. |
Web Crawler - Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/) |
163 |
8. |
Web Crawler - Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/) |
5 |
9. |
Web Crawler - www.integromedb.org/Crawler |
20 |
10. |
Web Crawler - Willow Internet Crawler by Twotrees V2.1 |
2 |
11. |
Web Crawler - WebZIP/x.x (http://www.spidersoft.com) |
1 |
12. |
Web Crawler - WebTarantula.com Crawler |
32 |
13. |
Web Crawler - WebCrawler |
8 |
14. |
Web Crawler - WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
463 |
15. |
Web Crawler - WebAlta Crawler/1.3.34 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
361 |
16. |
Web Crawler - WebAlta Crawler/1.3.23 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
245 |
17. |
Web Crawler - W3C_Validator/1.654 |
1 |
18. |
Web Crawler - W3C_Validator/1.3 http://validator.w3.org/services |
3 |
19. |
Web Crawler - Virusdie crawler/3.0 |
252 |
20. |
Web Crawler - VelenPublicWebCrawler/0.1 |
1 |
21. |
Web Crawler - VelenPublicWebCrawler (velen.io) |
19 |
22. |
Web Crawler - UXCrawlerBot |
4 |
23. |
Web Crawler - Utopia WebWasher 3.0 |
2 |
24. |
Web Crawler - User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
46 |
25. |
Web Crawler - UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de) |
1 |
26. |
Web Crawler - UCMore Crawler App |
3 |
27. |
Web Crawler - TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html) |
3156 |
28. |
Web Crawler - TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html) |
2967 |
29. |
Web Crawler - TurnitinBot (https://turnitin.com/robot/crawlerinfo.html) |
1 |
30. |
Web Crawler - Teemer (NetSeer, Inc.; http://www.netseer.com/crawler.html; crawler@netseer.com) |
1 |
31. |
Web Crawler - Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com) |
2 |
32. |
Web Crawler - spotinfluence/Nutch-1.4 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com) |
1 |
33. |
Web Crawler - spiderman |
953 |
34. |
Web Crawler - Social Media Crawler using your Home URL on Twitter,Facebook,Myspace,Linkedin by ProfileCapture - contact profilecapture@gmail.com to report any problems with my crawling. http://profilecapture.com |
1 |
35. |
Web Crawler - slurp, yahoo! slurp, slurp/2.0, inktomi slurp, slurp.so/1.0 |
1 |
36. |
Web Crawler - SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info at similarpages dot com) |
9 |
37. |
Web Crawler - ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com) |
6 |
38. |
Web Crawler - SheenBot/SheenBot-1.0.4 (Sheen web crawler) |
2 |
39. |
Web Crawler - SheenBot/SheenBot-1.0.3 (Sheen web crawler) |
3 |
40. |
Web Crawler - SheenBot/SheenBot-1.0.2 (Sheen web crawler) |
4 |
41. |
Web Crawler - SheenBot/SheenBot-1.0.1 (Sheen web crawler) |
4 |
42. |
Web Crawler - SheenBot/SheenBot-1.0.0 (Sheen web crawler) |
1 |
43. |
Web Crawler - ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com) |
542 |
44. |
Web Crawler - SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) |
5 |
45. |
Web Crawler - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
5718 |
46. |
Web Crawler - rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)/Nutch-1.13 |
2 |
47. |
Web Crawler - rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com) |
8 |
48. |
Web Crawler - rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org) |
1 |
49. |
Web Crawler - RMA/1.0 (compatible; RealMedia) |
2 |
50. |
Web Crawler - renlifangbot/1.0 (+http://search.msn.com/msnbot.htm) |
57 |
51. |
Web Crawler - REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project) |
1 |
52. |
Web Crawler - R6_FeedFetcher_(www.radian6.com/crawler) |
1 |
53. |
Web Crawler - Quick-Crawler (+https://www.scrapinghub.com/) |
4 |
54. |
Web Crawler - Python-urllib/2.7 (+http://example.com/crawler.html) |
5 |
55. |
Web Crawler - ppffpbuumkxcfHakxcmdhont wrma x |
1 |
56. |
Web Crawler - Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (for mobile crawler) ) |
4 |
57. |
Web Crawler - NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx) |
348 |
58. |
Web Crawler - netEstate NE Crawler (+http://www.website-datenbank.de/) |
14797 |
59. |
Web Crawler - netEstate NE Crawler (+http://www.sengine.info/) |
7435 |
60. |
Web Crawler - MSR-ISRCCrawler |
18 |
61. |
Web Crawler - msnbot/2.0b (+http://search.msn.com/msnbot.htm)._ |
19725 |
62. |
Web Crawler - msnbot/2.0b (+http://search.msn.com/msnbot.htm) |
21479 |
63. |
Web Crawler - msnbot/2.0b |
9 |
64. |
Web Crawler - msnbot/1.1 (+http://search.msn.com/msnbot.htm) |
12779 |
65. |
Web Crawler - msnbot/1.0 (+http://search.msn.com/msnbot.htm) |
6558 |
66. |
Web Crawler - msnbot/1.0 ( http://search.msn.com/msnbot.htm) |
1 |
67. |
Web Crawler - msnbot/0.01 (+http://search.msn.com/msnbot.htm) |
530 |
68. |
Web Crawler - msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) |
739 |
69. |
Web Crawler - msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) |
3351 |
70. |
Web Crawler - Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru) |
2 |
71. |
Web Crawler - Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0 |
7 |
72. |
Web Crawler - Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0 |
8 |
73. |
Web Crawler - Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
2 |
74. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
89854 |
75. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://abwc.iespana.es/last.php) |
2 |
76. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://abwc.iespana.es/forum/index.php) |
2 |
77. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) |
25933 |
78. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) |
2 |
79. |
Web Crawler - Mozilla/5.0 (compatible; WebHistoryBot/1.2.1 IS NOT SE bot like Googlebot/2.1; +http://www.google.com/bot.html,Yahoo! Slurp or Bingbot) |
3 |
80. |
Web Crawler - Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io) |
4 |
81. |
Web Crawler - Mozilla/5.0 (compatible; TestCrawler) |
9 |
82. |
Web Crawler - Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/) |
57403 |
83. |
Web Crawler - Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot) |
1 |
84. |
Web Crawler - Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler) |
1 |
85. |
Web Crawler - Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/) |
3 |
86. |
Web Crawler - Mozilla/5.0 (compatible; PathDefender/1.0; +http://www.pathdefender.com/help/crawler) |
1 |
87. |
Web Crawler - Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/) |
350 |
88. |
Web Crawler - Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/) |
148 |
89. |
Web Crawler - Mozilla/5.0 (compatible; NOT Googlebot/2.1; +http://www.google.com/bot.html) |
4 |
90. |
Web Crawler - Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru) |
8 |
91. |
Web Crawler - Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com) |
38 |
92. |
Web Crawler - Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com) |
101 |
93. |
Web Crawler - Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler) |
388697 |
94. |
Web Crawler - Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler) |
400 |
95. |
Web Crawler - Mozilla/5.0 (compatible; heritrix/3.0.0 +http://dataclip.com/crawler.html) |
7 |
96. |
Web Crawler - Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) |
114 |
97. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html) |
25 |
98. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/vb/index.php) |
2 |
99. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/last.php) |
1 |
100. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/forum/index.php) |
6 |
101. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/blog/last.php) |
1 |
102. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
4212061 |
103. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html |
3 |
104. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com) |
3 |
105. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io) |
126 |
106. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) |
21 |
107. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1 +http://www.googlebot.com/bot.html) |
1 |
108. |
Web Crawler - Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/) |
17 |
109. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler) |
4 |
110. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html) |
3 |
111. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler) |
2 |
112. |
Web Crawler - Mozilla/5.0 (compatible; fr-crawler/1.1) |
5099 |
113. |
Web Crawler - Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/legal/crawler) |
39 |
114. |
Web Crawler - Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/) |
496 |
115. |
Web Crawler - Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org) |
12142 |
116. |
Web Crawler - Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com) |
1 |
117. |
Web Crawler - Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions. |
15 |
118. |
Web Crawler - Mozilla/5.0 (compatible; ca-crawler/1.0) |
2025 |
119. |
Web Crawler - Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) |
83392 |
120. |
Web Crawler - Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com) |
6 |
121. |
Web Crawler - Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com) |
3 |
122. |
Web Crawler - Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler) |
17 |
123. |
Web Crawler - Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler) |
100186 |
124. |
Web Crawler - Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; ) |
2 |
125. |
Web Crawler - Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620 |
5 |
126. |
Web Crawler - Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/) |
24699 |
127. |
Web Crawler - Mozilla/4.7 (compatible; OffByOne; Windows 2000) Webster Pro V3.4 |
30 |
128. |
Web Crawler - Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) |
30 |
129. |
Web Crawler - Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) |
1 |
130. |
Web Crawler - Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT) |
7933 |
131. |
Web Crawler - Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com) |
26 |
132. |
Web Crawler - Mozilla/2.0 (compatible; NEWT ActiveX; Win32) |
1 |
133. |
Web Crawler - MBCrawler/1.0 (https://monitorbacklinks.com) |
98 |
134. |
Web Crawler - MauiBot (crawler.feedback+wc@gmail.com) |
8886 |
135. |
Web Crawler - MauiBot (crawler.feedback+dc@gmail.com) |
464 |
136. |
Web Crawler - LSSRocketCrawler/1.0 LightspeedSystems |
231 |
137. |
Web Crawler - LivelapBot/0.2 (http://site.livelap.com/crawler) |
2 |
138. |
Web Crawler - LinksCrawler 0.1beta |
3 |
139. |
Web Crawler - librabot/1.0 (+http://search.msn.com/msnbot.htm) |
52 |
140. |
Web Crawler - KSCrawler/Nutch-1.0 (http://www.kindsight.net/en/kscrawler; crawler@kindsight.net) |
2 |
141. |
Web Crawler - kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler) |
57 |
142. |
Web Crawler - Jayde Crawler. http://www.jayde.com |
3 |
143. |
Web Crawler - Jayde crawler |
3 |
144. |
Web Crawler - IP-Guide.com Crawler/1.0 (https://ip-guide.com) |
1 |
145. |
Web Crawler - ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company.; http://www.ilial.com/crawler; crawl@ilial.com) |
2 |
146. |
Web Crawler - ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html) |
2 |
147. |
Web Crawler - ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) |
2 |
148. |
Web Crawler - ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) |
2 |
149. |
Web Crawler - ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) |
3 |
150. |
Web Crawler - ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp) |
5 |
151. |
Web Crawler - ia_archiver/8.5 (Windows NT 7.2; ) |
1 |
152. |
Web Crawler - ia_archiver/8.2 (Windows NT 4.4; ) |
1 |
153. |
Web Crawler - ia_archiver/8.1 (Windows NT 5.6; ) |
1 |
154. |
Web Crawler - ia_archiver/7.9 (Windows XP 1.1; ) |
1 |
155. |
Web Crawler - ia_archiver/7.9 (Windows NT 6.3; ) |
1 |
156. |
Web Crawler - ia_archiver/7.3 (Windows 6.2; ) |
1 |
157. |
Web Crawler - ia_archiver/5.3 (Linux 1.9; ) |
1 |
158. |
Web Crawler - ia_archiver/5.1 (Windows NT 1.8; ) |
1 |
159. |
Web Crawler - ia_archiver/5.0 (OSX 6.4; ) |
1 |
160. |
Web Crawler - ia_archiver/4.6 (Windows NT 2.1; ) |
1 |
161. |
Web Crawler - ia_archiver/4.6 (OSX 6.2; ) |
1 |
162. |
Web Crawler - ia_archiver/4.5 (Windows NT 1.6; ) |
1 |
163. |
Web Crawler - ia_archiver/4.4 (OSX 6.7; ) |
1 |
164. |
Web Crawler - ia_archiver/3.9 (OSX 4.7; ) |
1 |
165. |
Web Crawler - ia_archiver/3.7 (Windows XP 3.5; ) |
1 |