hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Writing the WAL to a different filesystem from the HFiles
Date Wed, 28 Dec 2011 19:45:40 GMT
> HFile store wants more of an object store with full object reads and writes.

No, definitely not full object reads, we use HDFS positioned reads, which allow us to request,
within a gigabyte plus store file, much smaller byte ranges (e.g. 64 KB), and receive back
only the requested data. We can "seek" around the file.

> >  > Is this a reasonable thing to attempt? Or am I out of my mind?
> >  Check back in this mailing list archive for opinions on hbase atop s3.

Aside from several IMHO showstopper performance problems, the shortest answer is HBase often
wants to promptly read back store files it has written, and S3 is too eventual often enough
(transient 404s or 500s) to preclude reliable operation.

Best regards,

       - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)


----- Original Message -----
> From: Jagane Sundar <jagane@sundar.org>
> To: "dev@hbase.apache.org" <dev@hbase.apache.org>
> Cc: 
> Sent: Wednesday, December 28, 2011 10:42 AM
> Subject: RE: Writing the WAL to a different filesystem from the HFiles
> 
> Hello Mikael,
> 
> I am fairly new to HBase, so correct me if I am wrong - it seems to me that the 
> WAL and the HFile store impose distinctly different requirements upon the 
> underlying storage. The WAL really wants Posix semantics (append, flush, etc.) 
> in a distributed filesystem whereas the HFile store wants more of an object 
> store with full object reads and writes.
> 
> Adding this configuration option, or better yet, making the WAL fully pluggable 
> as described in https://issues.apache.org/jira/browse/HBASE-4529, opens up a lot 
> more storage architectures to HBase.
> 
> Thanks,
> Jagane
> ________________________________________
> From: Mikael Sitruk [mikael.sitruk@gmail.com]
> Sent: Wednesday, December 28, 2011 7:29 AM
> To: dev@hbase.apache.org
> Subject: Re: Writing the WAL to a different filesystem from the HFiles
> 
> What is the advantage of such configuration?
> Mikael.s
> On Dec 28, 2011 1:10 AM, "Stack" <stack@duboce.net> wrote:
> 
>>  On Mon, Dec 26, 2011 at 8:06 PM, Jagane Sundar <jagane@sundar.org> 
> wrote:
>>  > Is this a reasonable thing to attempt? Or am I out of my mind?
>>  >
>> 
>>  Check back in this mailing list archive for opinions on hbase atop s3.
>>  St.Ack
>> 
> 

Mime
View raw message