nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doğacan Güney (JIRA) <>
Subject [jira] Commented: (NUTCH-444) Possibly use a different library to parse RSS feed for improved performance and compatibility
Date Tue, 13 Feb 2007 09:00:26 GMT


Doğacan Güney commented on NUTCH-444:

Hi nutch.newbie,

Can you mail me a list of the failing atom urls(or if you can isolate it to a couple of atom
feeds - post it here)? If crawl fails in dedup or index(and not parse), this means that there
is a bug in NUTCH-443.

>One thing that I am a bit confused is the following. Lets say you have a feed with 5 items
i.e. 5 title 5 desc shouldn't the search >result i.e. if you do shoot out
6 results? 1 for the main feed page which is the actual feed URL and the other 5 for >the
5 items.. Currently I get only 1 search result which is the feed URL.
>Do I need to do 2 round of fetch? Cos things are getting parsed correctly.. maybe its
because I don't have the indexing plugin i.e >index-feed? no? I know we will work on it
after Nutch-443 is done..but I want to get a clarification..thats all :-) Cheers

I wouldn't worry about any shortcomings yet. Pretty much anything other blog search engines
does can also be done in Nutch. (And, yes, what you mention can be done too.)

> Possibly use a different library to parse RSS feed for improved performance and compatibility
> ---------------------------------------------------------------------------------------------
>                 Key: NUTCH-444
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: Renaud Richardet
>            Priority: Minor
>             Fix For: 0.9.0
>         Attachments: parse-feed-v2.tar.bz2, parse-feed.tar.bz2
> As discussed by Nutch Newbie, Gal, and Chris on NUTCH-443, the current library (feedparser)
has the following issues:
> - OutOfMemory when parsing > 100k feeds, since it has to convert the feed to jdom
> - no support for Atom 1.0
> - there has been no development in the last year
> Alternatives are:
> - Rome 
> - Informa
> - custom implementation based on Stax
> - ??

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message