Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of
 hrishikesh_agashe@persistent.co.in designates 202.54.11.87 as permitted
 sender)
From: Hrishikesh Agashe <hrishikesh_agashe@persistent.co.in>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Tue, 10 Nov 2009 15:26:33 +0530
Subject: Lucene + Hadoop
Thread-Topic: Lucene + Hadoop
Thread-Index: Acph7BV2h6H4f+uSRNG0dqFYC9Jtig==
Message-ID: 
 <E3137F97B0A0804194F369F4E7C3B71929AF5E59D8@EXCHANGE.persistent.co.in>
Accept-Language: en-US, en-IN
Content-Language: en-US
acceptlanguage: en-US, en-IN
Content-Type: multipart/alternative;
	boundary="_000_E3137F97B0A0804194F369F4E7C3B71929AF5E59D8EXCHANGEpersi_"
MIME-Version: 1.0

--_000_E3137F97B0A0804194F369F4E7C3B71929AF5E59D8EXCHANGEpersi_
Content-Type: text/plain; charset="us-ascii"
content-transfer-encoding: quoted-printable

Hi,

I am trying to use Hadoop for Lucene index creation. I have to create multip=
le indexes based on contents of the files (i.e. if author is "hrishikesh", i=
t should be added to a index for "hrishikesh". There has to be a separate in=
dex for every author). For this, I am keeping multiple IndexWriter open for=
 every author and maintaining them in a hashmap in map() function. I parse i=
ncoming file and if I see author is one for which I already have opened a In=
dexWriter, I just add this file in that index, else I create a new IndesWrit=
er for new author. As authors might run into thousands, I am closing IndexWr=
iter and clearing hashmap once it reaches a certain threshold and starting a=
ll over again. There is no reduced function.

Does this logic sound correct? Is there any other way of implementing this r=
equirement?

--Hrishi

DISCLAIMER=0A=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A=
This e-mail may contain privileged and confidential information which is the=
 property of Persistent Systems Ltd. It is intended only for the use of the=
 individual or entity to which it is addressed. If you are not the intended=
 recipient, you are not authorized to read, retain, copy, print, distribute=
 or use this message. If you have received this communication in error, plea=
se notify the sender and delete all copies of this message. Persistent Syste=
ms Ltd. does not accept any liability for virus infected mails.

--_000_E3137F97B0A0804194F369F4E7C3B71929AF5E59D8EXCHANGEpersi_--