Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of rohitkelkar@gmail.com
 designates 209.85.210.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <4F2650AC.20901@gmail.com>
References: 
 <CALCVZZs0wX5m_+TzJANxy-7mZvCLjXXWnXiqat8u7ME8v+PxJA@mail.gmail.com>
	<4F2281B7.3060509@gmail.com>
	<CALCVZZs=iPTc9FPkjrXiwhshMYz-Wj=q4WFACZBRW214+MwSwg@mail.gmail.com>
	<CADcMMgGq3G_njjb-kDTRjtg+LoXeWMZuo6nOXRSgi_6-92mnKQ@mail.gmail.com>
	<CALCVZZvw7n3A73c2=bgdL7D3i0x82T=voRstwz1LGhtf_jy9SA@mail.gmail.com>
	<4F2650AC.20901@gmail.com>
Date: Mon, 30 Jan 2012 13:49:51 +0530
Message-ID: 
 <CALCVZZs6PYnJpcM86TfHzo2x8STeUOwFdU=VHfbuHOfFkXNRVw@mail.gmail.com>
Subject: Re: advice needed on storing large objects on hdfs
From: Rohit Kelkar <rohitkelkar@gmail.com>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Ioan, Sorry for messing up your name. Your strategy sounds
interesting. I will try that out and post the results/problems if and
when ...

- Rohit Kelkar

On Mon, Jan 30, 2012 at 1:41 PM, Ioan Eugen Stan <stan.ieugen@gmail.com> wr=
ote:
> Pe 30.01.2012 09:53, Rohit Kelkar a scris:
>
>> Hi Stack,
>> My problem is that I have large number of smaller objects and a few
>> larger objects. My strategy is to store smaller objects (size< =A05MB)
>> in hbase and larger objects (size> =A05MB) on hdfs. And I also want to
>> run MapReduce tasks on those objects. Loan suggested that I should put
>> all objects in a MapFile/SequenceFile on hdfs and insert in to hbase
>> the reference of the object stored in the file. Now if I run a
>> mapreduce task, my mapper would be run locally wrt the object
>> references and not the actual dfs block where the object resides.
>>
>> - Rohit Kelkar
>
>
> Hi Rohit,
>
> First my name is Ioan (with i), second. It's a tricky question. If you ru=
n
> MapReduce with input from HBase you will have data locality for HBase dat=
a
> and not from the data in your SequenceFiles. You could get data locality
> from those if you perform a pre-setup job that scans HBase and builds a l=
ist
> of files to process and then runs another MR job on Hadoop targeting the
> SequenceFile. I think you can find ways to optimize the pre-process step =
to
> be fast.
>
> The set-up that I described is more suitable for situations when you need=
 to
> stream data that it's larger then a HBase is recommended to handle like
> mailboxes with large attachements. I'm planning to implement it soon in
> Apache James's HBase mailbox implementation to deal with large inboxes.
>
> Cheers,
>
>
> --
> Ioan Eugen Stan
> http://ieugen.blogspot.com