排名前50个开源的Web爬虫
项目名 | 开发语言 | 平台 |
---|---|---|
Heritrix | Java | Linux |
Nutch | Java | Cross-platform |
Scrapy | Python | Cross-platform |
DataparkSearch | C++ | Cross-platform |
GNU Wget | C | Linux |
GRUB | C#, C, Python, Perl | Cross-platform |
ht://Dig | C++ | Unix |
HTTrack | C/C++ | Cross-platform |
ICDL Crawler | C++ | Cross-platform |
mnoGoSearch | C | Windows |
Norconex HTTP Collector | Java | Cross-platform |
Open Source Server | C/C++, Java PHP | Cross-platform |
PHP-Crawler | PHP | Cross-platform |
YaCy | Java | Cross-platform |
WebSPHINX | Java | Cross-platform |
WebLech | Java | Cross-platform |
Arale | Java | Cross-platform |
JSpider | Java | Cross-platform |
HyperSpider | Java | Cross-platform |
Arachnid | Java | Cross-platform |
Spindle | Java | Cross-platform |
Spider | Java | Cross-platform |
LARM | Java | Cross-platform |
Metis | Java | Cross-platform |
SimpleSpider | Java | Cross-platform |
Grunk | Java | Cross-platform |
CAPEK | Java | Cross-platform |
Aperture | Java | Cross-platform |
Smart and Simple Web Crawler | Java | Cross-platform |
Web Harvest | Java | Cross-platform |
Aspseek | C++ | Linux |
Bixo | Java | Cross-platform |
crawler4j | Java | Cross-platform |
Ebot | Erland | Linux |
Hounder | Java | Cross-platform |
Hyper Estraier | C/C++ | Cross-platform |
OpenWebSpider | C#, PHP | Cross-platform |
Pavuk | C | Lunix |
Sphider | PHP | Cross-platform |
Xapian | C++ | Cross-platform |
Arachnode.net | C# | Windows |
Crawwwler | C++ | Java |
Distributed Web Crawler | C, Java, Python | Cross-platform |
iCrawler | Java | Cross-platform |
pycreep | Java | Cross-platform |
Opese | C++ | Linux |
Andjing | Java | |
Ccrawler | C# | Windows |
WebEater | Java | Cross-platform |
JoBo | Java | Cross-platform |