hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: ZK rethink?
Date Sun, 12 Apr 2009 19:44:45 GMT

I ran an initial test of HBase on top of KFS -- using the
Hadoop FS abstraction layer, which is the only current
integration option -- and ran into some trouble under load.
I tested with kfs-0.2.3 on 0.19.1 plus the patch for
HADOOP-5292. It may have just been a function of the scale
of the deployment (all localhost :-) ) but the chunkserver
after a time began to expand its address space by gigs,
beyond 40 GB of address space in one instance before I
killed it. I may play around with it again on a testbed of
several nodes at some point. 

   - Andy

> From: Chad Walters
> Subject: Re: ZK rethink?
> Date: Sunday, April 12, 2009, 7:43 AM
> My understanding is that Quantcast was running both HDFS and
> KFS in parallel since they didn't  fully trust either
> one. Can anyone confirm or deny this? Have they switched
> over fully to using KFS?
> KFS seemed interesting but, given that more development
> effort is directed at HDFS, it didn't seem worth
> pursuing. Of course, as you point out, HDFS has been
> slow/resistant to implementing some features important for
> HBase that KFS supports out of the box.
> As things currently stand, it is unlikely that folks from
> the Powerset team will lead any investigation into KFS.
> However, others are welcome to spend some time digging into
> it and seeing how promising the prospect is.
> I believe that KFS has an implementation of the Hadoop File
> System interface. It seems like someone who was interested
> and motivated could hook up KFS under HBase that way to do
> some basic testing. Perhaps some benchmarking with KFS would
> turn up some numbers that could be used to spur some changes
> at HDFS...
> Doing deeper integration with KFS that avoided the Hadoop
> FileSystem abstraction seems like it would be a mistake (but
> I could be convinced otherwise). Proposing possible
> extensions to the Hadoop FileSystem interface might be more
> fruitful.
> Chad


View raw message