Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of kevin.odell@cloudera.com
 designates 209.85.215.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALte62yBB0Xvqx+9_xf0Dpda5jvakhD22+b6hm0G4riA=oMrCQ@mail.gmail.com>
References: 
 <CANpfeLzRKXsXj3n160g6XDYaFUkesxqikNHOYK7-QjbcbW+xug@mail.gmail.com>
	<CALte62xgU6_RfYW0rfx2Bqb2=WZZESeHzkGa4sWvf7ZBuKFrfA@mail.gmail.com>
	<CANpfeLx_ajjZdRZHRnUSvBoC3ekKBKz+mda8C9bf5moU9_miRg@mail.gmail.com>
	<DC5EBE7F3610EB4CA5C7E92D76873E1518481D86CB@exchange2007.carrieriq.com>
	<CALte62yBB0Xvqx+9_xf0Dpda5jvakhD22+b6hm0G4riA=oMrCQ@mail.gmail.com>
Date: Sat, 4 Jan 2014 16:19:25 -0500
Message-ID: 
 <CAGngS9dH1mTS+s0hv+44N6zHvzPdrH4gc59QqoKLoLmWbZs-NA@mail.gmail.com>
Subject: Re: Hbase Performance Issue
From: "Kevin O'dell" <kevin.odell@cloudera.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=047d7b33dc4ac2b30f04ef2b9430

--047d7b33dc4ac2b30f04ef2b9430
Content-Type: text/plain; charset=ISO-8859-1

Have you tried writing out an hfile and then bulk loading the data?
On Jan 4, 2014 4:01 PM, "Ted Yu" <yuzhihong@gmail.com> wrote:

> bq. Output is written to either Hbase
>
> Looks like Akhtar wants to boost write performance to HBase.
> MapReduce over snapshot files targets higher read throughput.
>
> Cheers
>
>
> On Sat, Jan 4, 2014 at 12:55 PM, Vladimir Rodionov
> <vrodionov@carrieriq.com>wrote:
>
> > You cay try MapReduce over snapshot files
> > https://issues.apache.org/jira/browse/HBASE-8369
> >
> > but you will need to patch 0.94.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: vrodionov@carrieriq.com
> >
> > ________________________________________
> > From: Akhtar Muhammad Din [akhtar.mdin@gmail.com]
> > Sent: Saturday, January 04, 2014 12:44 PM
> > To: user@hbase.apache.org
> > Subject: Re: Hbase Performance Issue
> >
> > im  using CDH 4.5:
> > Hadoop:  2.0.0-cdh4.5.0
> > HBase:   0.94.6-cdh4.5.0
> >
> > Regards
> >
> >
> > On Sun, Jan 5, 2014 at 1:24 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > What version of HBase / hdfs are you running with ?
> > >
> > > Cheers
> > >
> > >
> > >
> > > On Sat, Jan 4, 2014 at 12:17 PM, Akhtar Muhammad Din
> > > <akhtar.mdin@gmail.com>wrote:
> > >
> > > > Hi,
> > > > I have been running a map reduce job that joins 2 datasets of 1.3
> and 4
> > > GB
> > > > in size. Joining is done at reduce side. Output is written to either
> > > Hbase
> > > > or HDFS depending upon configuration. The problem I am having is that
> > > Hbase
> > > > takes about 60-80 minutes to write the processed data, on the other
> > hand
> > > > HDFS takes only 3-5 mins to write the same data. I really want to
> > improve
> > > > the Hbase speed and bring it down to 1-2 min.
> > > >
> > > > I am using amazon EC2 instances, launched a cluster of size 3 and
> later
> > > 10,
> > > > have tried both c3.4xlarge and c3.8xlarge instances.
> > > >
> > > > I can see significant increase in performance while writing to HDFS
> as
> > i
> > > > use cluster with more nodes, having high specifications, but in the
> > case
> > > of
> > > > Hbase there was no significant change in performance.
> > > >
> > > > I have been going through different posts, articles and have read
> Hbase
> > > > book to solve the Hbase performance issue but have not been able to
> > > succeed
> > > > so far.
> > > > Here are the few things i have tried out so far:
> > > >
> > > > *Client Side*
> > > > - Turned off writing to WAL
> > > > - Experimented with write buffer size
> > > > - Turned off auto flush on table
> > > > - Used cache, experimented with different sizes
> > > >
> > > >
> > > > *Hbase Server Side*
> > > > - Increased region servers heap size to 8 GB
> > > > - Experimented with handlers count
> > > > - Increased Memstore flush size to 512 MB
> > > > - Experimented with hbase.hregion.max.filesize, tried different sizes
> > > >
> > > > There are many other parameters i have tried out following the
> > > suggestions
> > > > from  different sources, but nothing worked so far.
> > > >
> > > > Your help will be really appreciated.
> > > >
> > > > --
> > > > Regards
> > > > Akhtar Muhammad Din
> > > >
> > >
> >
> >
> >
> > --
> > Regards
> > Akhtar Muhammad Din
> >
> > Confidentiality Notice:  The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited.  If you have received this message in error,
> please
> > immediately notify the sender and/or Notifications@carrieriq.com and
> > delete or destroy any copy of this message and its attachments.
> >
>

--047d7b33dc4ac2b30f04ef2b9430--