nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doğacan Güney (JIRA) <>
Subject [jira] Commented: (NUTCH-443) allow parsers to return multiple Parse object, this will speed up the rss parser
Date Tue, 27 Feb 2007 20:11:05 GMT


Doğacan Güney commented on NUTCH-443:

Hi Andrzej,

> * in my opinion it's easier to add missing CrawlDatum's (with correctly set fetch time)
for the new urls to the 
> output rather than work-around this by passing around the fetch time in metadata, and
then again 
> compensating in Indexer and CrawlDbReducer for the lack of these fetchDatum-s.

I guess what you mean is pushing STATUS_FETCH_SUCCESS datums to crawl_parse, right? I can
probably do this by changing ParseImpl and adding a new boolean to it that indicates whether
it is fetched or freshly generated in parse.

> * in Fetcher / Fetcher2 you don't pass the signature in case when there is no valid Parse
output, but in the 
> current versions of Fetchers the signature is still calculated and passed in datum.setSignature()
(which ends 
> up in crawl_fetch). 

OK, I will fix it.

> * using a generic Map<String, Parse> is IMHO inappropriate, as I indicated earlier,
especially since this Map > requires special post-processing in ParseUtil.processParseMap
- and what would happen if I didn't use 
> ParseUtil? I think this calls for a special-purpose class (ParseResult?), which would
encapsulate this 
> behavior without exposing it to its users (or even worse - allowing users to bypass it).
This class would also > help us to avoid somewhat ugly "convenience" methods in ParseStatus
and ParseImpl - these details would > be hidden in one of the constructors of ParseResult.

> * I'm also not sure why we use Map<String, Parse> and not Map<Text, Parse>,
since in all further 
> processing we need to create Text objects ...

If we are going with a special-purpose class, there is one more thing I would like to change.
Consider the case of a zip archive with url that contains two files
spam.txt, egg.txt. After parsing this you will return something like <key1, parse of spam.txt>,
<key2, parse of egg.txt> and perhaps <original_url, who knows what>. 

Now, whatever key1 and key2 is, they are not really urls to be fetched. So I want to add another
fetch and db status (let's call them STATUS_FETCH_FAKE and STATUS_DB_FAKE). During parse key1
and key2 will be written with FETCH_FAKE, and updatedb will write them as DB_FAKE to crawldb.
Nutch will still index things with FAKE status, but generate will never generate them to be
fetched. And updatedb will never change their status to DB_UNFETCHED(since, as I said before,
they can't be fetched).

So, ParseResult will contain a group of <'real' url, parse> and a group of <'phony'
url, parse> pairs.

What do you think?

> * the new section in HtmlParseFilters breaks the loop on encountering the first error,
and leaves the parse 
> results incompletely filtered. It should simply continue - the result is an aggregation
of more or less 
> independent documents that are parsed on their own. 

This is the same as the old behavior. Why change it? (There was a bug there, but I fixed in
one of the newer patches)

> * the comment about redirects in is misplaced - I think this contract should
be both defined and > enforced in the Fetcher. 


> And finally, I think this is a significant change in the way how content parsers work
with the rest of the 
> framework, so we should wait with this patch after the 0.9 release - and we should push
0.9 out of the door 
> really soon ...

Anything to get 0.9 out of the door:)

I will send an updated patch that fixes 1,2 and 4 (and 5 if I am missing something there)
tomorrow unless someone beats me to it. I want to hear what others think on 3 before doing

Thanks for your review and comments.

> allow parsers to return multiple Parse object, this will speed up the rss parser
> --------------------------------------------------------------------------------
>                 Key: NUTCH-443
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: Renaud Richardet
>         Assigned To: Chris A. Mattmann
>            Priority: Minor
>             Fix For: 0.9.0
>         Attachments: NUTCH-443-draft-v1.patch, NUTCH-443-draft-v2.patch, NUTCH-443-draft-v3.patch,
NUTCH-443-draft-v4.patch, NUTCH-443-draft-v5.patch, NUTCH-443-draft-v6.patch, NUTCH-443-draft-v7.patch,
NUTCH-443.022507.patch.txt, parse-map-core-draft-v1.patch, parse-map-core-untested.patch,
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser can
return multiple parse objects, that will all be indexed separately. Advantage: no need to
fetch all feed-items separately.
> see the discussion at

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message