hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Meil <doug.m...@explorysmedical.com>
Subject Re: Shared HDFS for HBase and MapReduce
Date Wed, 06 Jun 2012 21:14:15 GMT

Regarding locality, it's not just Lars' stuff, it's in the RefGuide (see
section 9.7.3)Š


re:  "You will still be reading/writing over the network"

This is definitely true as far as writes go because of the replicas (see
the RefGuide for why), although I disagree on the read portion unless
there is an exceptional case (which typically the result of an RS going

On 6/6/12 4:27 PM, "Atif Khan" <atif_ijaz_khan@hotmail.com> wrote:

>Thanks Amandeep!
>I think what I was saying that we are trying to support both types of
>workloads.  That is realtime transactional workloads, and batch processing
>for data analysis.  The big question being if a single HDFS cluster should
>be shared between the two workflows.
>The point that you are trying to make (if I am understanding you
>is of data "Locality".
>/Amandeep Khurana - "Having a common HDFS cluster and using part of the
>nodes as HBase RS and part as the Hadoop TTs doesn't solve the problem of
>moving data from the HBase RS to the tasks you'll run as a part of your MR
>jobs if HBase is your source/sink. You will still be reading/writing over
>the network."
>When running MR jobs over HBase, data locality is provided by HBase
>see http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html,
>also HBase the Definitive Guide by Lars George page 298 MapReduce
>In other words, the computation will be exported to where the data is,
>therefore limiting the need to transfer data over the network.  Proper
>locality has a big impact on the overall performance.
>So I believe that a common HDFS cluster does not imply logical segregation
>between HBase RS and Hadoop TTs.  Therefore, your point seems in
>contradiction with Lars George's statement.
>View this message in context:
>Sent from the HBase - Developer mailing list archive at Nabble.com.

View raw message