nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dogacan Güney (JIRA) <j...@apache.org>
Subject [jira] Commented: (NUTCH-443) allow parsers to return multiple Parse object, this will speed up the rss parser
Date Thu, 08 Feb 2007 10:17:07 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12471260
] 

Dogacan Güney commented on NUTCH-443:
-------------------------------------

Ok, this is  the second attempt(sorry that I am sending patches in a frenzy, I will slow down
now). 

In the first patch, I just put Map<String, Parse> to FetcherOutput but that doesn't
work, 
since the keys are not necessarily ordered.

I mean;
Assume, we have two <key, Map<String, Parse>> pairs:
<"a.com", <"z.com", some_parse>> and <"b.com", <"b.com", some_other_parse>>
With the first patch, we would first get a.com (and thus write z.com) then get b.com (and
try to write b.com)
but this would fail since "b.com" < "z.com".

I completely removed FetcherOutput class. What it does can be done with wrapping the objects

in ObjectWritable. I know this is heavier but I couldn't think of another way around the issue
of ordering
of keys.

I tested this a bit with a small set of urls. Both parsing seperately and parsing during fetching
seems to work.

> allow parsers to return multiple Parse object, this will speed up the rss parser
> --------------------------------------------------------------------------------
>
>                 Key: NUTCH-443
>                 URL: https://issues.apache.org/jira/browse/NUTCH-443
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: Renaud Richardet
>            Priority: Minor
>             Fix For: 0.9.0
>
>         Attachments: parse-map-core-untested.patch
>
>
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser can
return multiple parse objects, that will all be indexed separately. Advantage: no need to
fetch all feed-items separately.
> see the discussion at http://www.nabble.com/RSS-fecter-and-index-individul-how-can-i-realize-this-function-tf3146271.html

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message