nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Nutch Wiki] Trivial Update of "ApacheConUs2009MeetUp" by KenKrugler
Date Mon, 03 Aug 2009 16:43:54 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The following page has been changed by KenKrugler:
http://wiki.apache.org/nutch/ApacheConUs2009MeetUp

------------------------------------------------------------------------------
  
  Below are some potential topics for discussion - feel free to add/comment.
  
- * Potential synergies between crawler projects - e.g. sharing robots.txt processing code.
+  * Potential synergies between crawler projects - e.g. sharing robots.txt processing code.
- * How to avoid end-user abuse - webmasters sometimes block crawlers because users configure
it to be impolite.
+  * How to avoid end-user abuse - webmasters sometimes block crawlers because users configure
it to be impolite.
- * Politeness vs. efficiency - various options for how to be considered polite, while still
crawling quickly.
+  * Politeness vs. efficiency - various options for how to be considered polite, while still
crawling quickly.
- * robots.txt processing - current problems with existing implementations
+  * robots.txt processing - current problems with existing implementations
- * Avoiding crawler traps - link farms, honeypots, etc.
+  * Avoiding crawler traps - link farms, honeypots, etc.
- * Parsing content - home grown, Neko/TagSoup, Tika, screen scraping
+  * Parsing content - home grown, Neko/TagSoup, Tika, screen scraping
- * Search infrastructure - options for serving up crawl results (Nutch, Solr, Katta, others?)
+  * Search infrastructure - options for serving up crawl results (Nutch, Solr, Katta, others?)
- * Testing challenges - is it possible to unit test a crawler?
+  * Testing challenges - is it possible to unit test a crawler?
- * Fuzzy classification - mime-type, charset, language.
+  * Fuzzy classification - mime-type, charset, language.
- * The future of Nutch, Droids, Heritrix, Bixo, etc.
+  * The future of Nutch, Droids, Heritrix, Bixo, etc.
- * Optimizing for types of crawling - intranet, focused, whole web.
+  * Optimizing for types of crawling - intranet, focused, whole web.
  

Mime
View raw message