nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doğacan Güney (JIRA) <>
Subject [jira] Commented: (NUTCH-443) allow parsers to return multiple Parse object, this will speed up the rss parser
Date Wed, 14 Feb 2007 17:34:05 GMT


Doğacan Güney commented on NUTCH-443:

> Hmm, actually this is an important question. I don't think FetcherOutput is persisted
anywhere, it's just an aggregate class to 
> keep things together before they hit the disk. I propose to leave a comment in MapWritable
like this "// code -123 was 
> reserved  for FetcherOutput - no longer in use". As for the class itself - again, since
it's not persisted we don't have to keep it 
> around, just remove it.

I implemented this approach in one of the earlier patches. The problem is that, the code in
MapWritable does this:

addIdEntry((byte) (-128 + CLASS_ID_MAP.size() + ++fIdCount), // ...

Now, I don't claim to understand the code perfectly but because of the "-128 + CLASS_ID_MAP.size()"
part I think CLASS_ID_MAP must have consecutive values always, so not having -123 breaks it.
IIRC, removing that line and running TestMapWritable fails.

> Sections in Fetcher.FetcherThread.output() and similar in Fetcher2 that output the data
need to be synchronized now - 
> output.collect() is no longer a single atomic operation. Perhaps it's better to leave
FetcherOutput after all?

This causes key ordering problems. See my admittedly-could-have-been-clearer 2nd comment.

Anyway, I am assumming that you are OK with removing ParseUtil.getFirstParseEntry and just
using Map.get?

> allow parsers to return multiple Parse object, this will speed up the rss parser
> --------------------------------------------------------------------------------
>                 Key: NUTCH-443
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: Renaud Richardet
>         Assigned To: Chris A. Mattmann
>            Priority: Minor
>             Fix For: 0.9.0
>         Attachments: NUTCH-443-draft-v1.patch, NUTCH-443-draft-v2.patch, NUTCH-443-draft-v3.patch,
NUTCH-443-draft-v4.patch, NUTCH-443-draft-v5.patch, NUTCH-443-draft-v6.patch, parse-map-core-draft-v1.patch,
parse-map-core-untested.patch, parsers.diff
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser can
return multiple parse objects, that will all be indexed separately. Advantage: no need to
fetch all feed-items separately.
> see the discussion at

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message