hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Faruk Berksöz <fberk...@gmail.com>
Subject Re: hbase insert performance test (from hbasemaster and regionservers)
Date Tue, 22 May 2012 20:21:17 GMT
Hello Michael ,

thank you for the hint


2012/5/22 Michael Segel <michael_segel@hotmail.com>

> Hi,
>
> Seems we just had someone talk about this just the other day...
>
> 1) 8GB of memory isn't enough to run both M/R and HBase.
> Ok, yes you can run it, however don't expect it to perform well.
>
> 2) You never want a user to run their own code from the cluster itself.
> Use an *edge* node.
>
> There's more, but you get the idea.
>
> On May 21, 2012, at 2:04 PM, Faruk Berksöz wrote:
>
> > Dear All,
> > we have 4 node in our cluster (1nn+dn,3 dn).
> > Hadoop dist. is cdh3u3.
> > Every node has 2 tb disk , 8 gb memory.
> > We are trying some insert performance test on hbase.
> > I have tried to insert 250.000 records from hbase master  (without
> thread),
> > that takes  5-7 sec.
> > But when I try ro insert from  any regionserver the same data (250.000)
>  ,
> > it takes longer  21 sec.
> > Is that normal ?
> >
> > any response would be appreciated..
> >
> >
> > my java code looks like:
> >            .........
> >            .........
> >            .........
> >            long start = System.currentTimeMillis();
> >            //DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd
> > HH:mm:ss");
> >            //Date sdate = new Date();
> >
> >            //LOG.info(" Start-Time :" + dateFormat.format(sdate));
> >            for (int i = 0; i < eachsize; i++) {
> >
> >
> >                put = new Put(String.format("%016x",
> > random.nextLong()).getBytes());
> >
> >                put.setWriteToWAL(false);
> >                //set column and their values
> >                addColumnAndValues();
> >                table.put(put);
> >
> >
> >            }
> >            elapsedTimeMillis = System.currentTimeMillis() - start;
> >            elapsedTimeSec = elapsedTimeMillis / 1000F;
> >            elapsedTimeSecInMemory = elapsedTimeSec ;
> >            LOG.info(" Elapsed-Time (sec) inMemory :" + elapsedTimeSec);
> >
> >            table.flushCommits();
> >            //disk write elapsed time
> >            elapsedTimeMillis = System.currentTimeMillis() - start;
> >            elapsedTimeSec = elapsedTimeMillis / 1000F;
> >            elapsedTimeSecDiskWrite   = elapsedTimeSec;
> >            LOG.info(" Elapsed-Time (sec) Disk Write:" +
> > elapsedTimeSec);
> >            .........
> >            .........
> >
> > My hbase-site.xml is looks like :
> > <configuration>
> >    <property>
> >        <name>hbase.rootdir</name>
> >        <value>hdfs://master.bigdata.com:54310/hbase</value>
> >    </property>
> >    <property>
> >        <name>hbase.cluster.distributed</name>
> >        <value>true</value>
> >    </property>
> >    <property>
> >        <name>hbase.zookeeper.quorum</name>
> >        <value>master.bigdata.com,slave1.bigdata.com,slave2.bigdata.com,
> > slave3.bigdata.com</value>
> >    </property>
> >    <property>
> >        <name>hbase.zookeeper.dns.interface</name>
> >        <value>eth0</value>
> >    </property>
> >    <property>
> >        <name>hbase.zookeeper.dns.nameserver</name>
> >        <value>10.10.10.1</value>
> >    </property>
> >
> >    <property>
> >        <name>hbase.regionserver.handler.count</name>
> >        <value>20</value>
> >    </property>
> >
> >    <property>
> >        <name>hbase.client.write.buffer</name>
> >        <value>5097152</value>
> >    </property>
> > </configuration>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message