Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of ning.li.00@gmail.com designates
 74.125.46.31 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=AzZfCrx9CIjbCabnLqzGWyZu+niXegAkKUMEuDaER9JlvMFc5uWIo8qo4gkeW/q5Yz
         eUlNoLcCaNPeHkzk+i6WUrQlf1IIylG1LsDQ3A+C8oMZ2EAgBME6x2AdqXSkP3eiDTzf
         QKfzBVUIvgZXIReqwyX14fWsDPE0F/lml3ez4=
MIME-Version: 1.0
In-Reply-To: <49BEC670.5080700@apache.org>
References: <c9b0d8bd0903122138u6d6c8babv53dbde1474ea309e@mail.gmail.com>
	 <8131791a0903122234g18eac67bqeb50a1dfcae361f2@mail.gmail.com>
	 <d32473880903131210t2b59cfe5q7f58bee55d764931@mail.gmail.com>
	 <9cfk56pfg81.fsf@rogue.ncsl.nist.gov>
	 <d32473880903161347p42ad2148q8c8f5681213b477c@mail.gmail.com>
	 <9cftz5tdwbz.fsf@rogue.ncsl.nist.gov>
	 <d32473880903161419l770bb40x7e00d8b0efccb163@mail.gmail.com>
	 <49BEC670.5080700@apache.org>
Date: Mon, 16 Mar 2009 19:21:35 -0400
Message-ID: <d32473880903161621g753e96d1nc0153dab402eb0b5@mail.gmail.com>
Subject: Re: Creating Lucene index in Hadoop
From: Ning Li <ning.li.00@gmail.com>
To: core-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

1 is good. But for 2:
  - Won't it have a security concern as well? Or is this not a general
local cache?
  - You are referring to caching in RAM, not caching in local FS,
right? In general, a Lucene index size could be quite large. We may
have to cache a lot of data to reach a reasonable hit ratio...

Cheers,
Ning


On Mon, Mar 16, 2009 at 5:36 PM, Doug Cutting <cutting@apache.org> wrote:
> Ning Li wrote:
>>
>> With
>> http://issues.apache.org/jira/browse/HADOOP-4801, however, it may
>> become feasible to search on HDFS directly.
>
> I don't think HADOOP-4801 is required. =A0It would help, certainly, but i=
t's
> so fraught with security and other issues that I doubt it will be committ=
ed
> anytime soon.
>
> What would probably help HDFS random access performance for Lucene
> significantly would be:
> =A01. A cache of connections to datanodes, so that each seek() does not
> require an open(). =A0If we move HDFS data transfer to be RPC-based (see,
> e.g., http://issues.apache.org/jira/browse/HADOOP-4386), then this will c=
ome
> for free, since RPC already caches connections. =A0We hope to do this for
> Hadoop 1.0, so that we use a single transport for all Hadoop's core
> operations, to simplify security.
> =A02. A local cache of read-only HDFS data, equivalent to kernel's buffer
> cache. =A0This might be implemented as a Lucene Directory that keeps an L=
RU
> cache of buffers from a wrapped filesystem, perhaps a subclass of
> RAMDirectory.
>
> With these, performance would still be slower than a local drive, but
> perhaps not so dramatically.
>
> Doug
>