Mailing-List: contact nutch-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: nutch-dev@lucene.apache.org
Message-ID: <1062435636.1138786264480.JavaMail.jira@ajax.apache.org>
Date: Wed, 1 Feb 2006 10:31:04 +0100 (CET)
From: "Andrzej Bialecki  (JIRA)" <jira@apache.org>
To: nutch-dev@incubator.apache.org
Subject: [jira] Commented: (NUTCH-192) meta data support for CrawlDatum
In-Reply-To: <718440621.1138666652937.JavaMail.jira@ajax.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

    [ http://issues.apache.org/jira/browse/NUTCH-192?page=comments#action_12364782 ] 

Andrzej Bialecki  commented on NUTCH-192:
-----------------------------------------

There is a very real hazard in the fact that we don't store the dictionary. Let's consider this example: two plugins invoke WritableName.setName() with different classes, ClassA and ClassB. We get the mapping ClassA -> 23, ClassB -> 24. The files written by these plugins use just the byte IDs, 23 and 24. The someone changes the config file, and plugins are initialized in a reversed order, so consequently we get ClassB -> 23, ClassA ->24. And now the plugins cannot read the files they created because of the wrong class returned from MapWritable ...

So, I'm still convinced that we need to save the dictionary. Unfortunately, for small amounts of metadata (typical use case) it blows up the on-disk size of MapWritable, which is why I thought using Strings would be cheaper ...

Other things: In the javadoc for MapWritable it should be mentioned that any Writable type that one is going to use needs to be first registered with the WritableName.setName(). Or perhaps the method could do it automatically, but then the IDs will be unpredictable, depending on the order of iteration (which leads to the problem described above).

Also, there is a bug in setName(): if you try adding the same mapping twice (which could happen in different places), the method should allocate just one ID for the class. As it is now, it will allocate new ID each time you call the method, even if the class name is the same. Just add this:

   public static synchronized void setName(Class writableClass, String name) {
     Object o = CLASS_TO_NAME.put(writableClass, name);
     NAME_TO_CLASS.put(name, writableClass);
     if (o != null) return; // already has an ID
    CLASS_TO_ID.put(writableClass, new Byte((byte)CLASS_TO_ID.size()));
    ID_TO_CLASS.put(new Byte((byte)ID_TO_CLASS.size()), writableClass);
   }

> meta data support for CrawlDatum
> --------------------------------
>
>          Key: NUTCH-192
>          URL: http://issues.apache.org/jira/browse/NUTCH-192
>      Project: Nutch
>         Type: Improvement
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>      Fix For: 0.8-dev
>  Attachments: metadata300106.patch, metadata310106.patch
>
> Supporting meta data in CrawlDatum would help to get a set of new nutch features realized and makes a lot possible to smaller special focused search engines.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira