Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of joey@cloudera.com designates
 209.85.161.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1313139409.51608.YahooMailNeo@web36102.mail.mud.yahoo.com>
References: <1313135635.27469.YahooMailNeo@web36108.mail.mud.yahoo.com>
	<CAA2hHbewGwA9J3Bs1AvjHK04_cK_8HPBewDvX8Bvt3Siqz5dsw@mail.gmail.com>
	<1313139409.51608.YahooMailNeo@web36102.mail.mud.yahoo.com>
Date: Fri, 12 Aug 2011 06:28:00 -0400
Message-ID: 
 <CAEGKEUPx=vJR5cpCk4jKTyw_PNxR-tzD2dEdrentZ7MFPE2=+w@mail.gmail.com>
Subject: Re: Hadoop--store a sequence file in distributed cache?
From: Joey Echeverria <joey@cloudera.com>
To: common-user@hadoop.apache.org, Sofia Georgiakaki <geosofie_tuc@yahoo.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

You can use any kind of format for files in the distributed cache, so
yes you can use sequence files. They should be faster to parse than
most text formats.

-Joey

On Fri, Aug 12, 2011 at 4:56 AM, Sofia Georgiakaki
<geosofie_tuc@yahoo.com> wrote:
> Thank you for the reply!
> In each map(), I need to open-read-close these files (more than 2 in the =
general case, and maybe up to 20 or more), in order to make some checks. Co=
nsidering the huge amount of data in the input, making all these file opera=
tions on HDFS will kill the performance!!! So I think it would be better to=
 store these files in distributed Cache, so that the whole process would be=
 more efficient -I guess this is the point of using Distributed Cache in th=
e first place!
>
> My question is, if I can store sequence files in distributed Cache and ha=
ndle them using e.g. the SequenceFile.Reader class, or if I should only kee=
p regular text files in distributed Cache and handle them using the usual j=
ava API.
>
> Thank you very much
> Sofia
>
> PS: The files have small size, a few KB to few MB maximum.
>
>
>
> ________________________________
> From: Dino Ke=C4=8Do <dino.keco@gmail.com>
> To: common-user@hadoop.apache.org; Sofia Georgiakaki <geosofie_tuc@yahoo.=
com>
> Sent: Friday, August 12, 2011 11:30 AM
> Subject: Re: Hadoop--store a sequence file in distributed cache?
>
> Hi Sofia,
>
> I assume that output of first job is stored on HDFS. In that case I would
> directly read file from Mappers without using distributed cache. If you p=
ut
> file into distributed cache that would add one more copy operation into y=
our
> process.
>
> Thanks,
> dino
>
>
> On Fri, Aug 12, 2011 at 9:53 AM, Sofia Georgiakaki
> <geosofie_tuc@yahoo.com>wrote:
>
>> Good morning,
>>
>> I would like to store some files in the distributed cache, in order to b=
e
>> opened and read from the mappers.
>> The files are produced by an other Job and are sequence files.
>> I am not sure if that format is proper for the distributed cache, as the
>> files in distr.cache are stored and read locally. Should I change the fo=
rmat
>> of the files in the previous Job and make them Text Files maybe and read
>> them from the Distr.Cache using tha simple Java API?
>> Or can I still handle them with the usual way we use sequence files, eve=
n
>> if they reside in the local directory? Performance is extremely importan=
t
>> for my project, so I don't know what the best solution would be.
>>
>> Thank you in advance,
>> Sofia Georgiakaki


--=20
Joseph Echeverria
Cloudera, Inc.
443.305.9434