hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Faruk Berksöz <fberk...@gmail.com>
Subject hbase insert performance test (from hbasemaster and regionservers)
Date Mon, 21 May 2012 19:04:49 GMT
Dear All,
we have 4 node in our cluster (1nn+dn,3 dn).
Hadoop dist. is cdh3u3.
Every node has 2 tb disk , 8 gb memory.
We are trying some insert performance test on hbase.
I have tried to insert 250.000 records from hbase master  (without thread),
that takes  5-7 sec.
But when I try ro insert from  any regionserver the same data (250.000)  ,
it takes longer  21 sec.
Is that normal ?

any response would be appreciated..


my java code looks like:
            .........
            .........
            .........
            long start = System.currentTimeMillis();
            //DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd
HH:mm:ss");
            //Date sdate = new Date();

            //LOG.info(" Start-Time :" + dateFormat.format(sdate));
            for (int i = 0; i < eachsize; i++) {


                put = new Put(String.format("%016x",
random.nextLong()).getBytes());

                put.setWriteToWAL(false);
                //set column and their values
                addColumnAndValues();
                table.put(put);


            }
            elapsedTimeMillis = System.currentTimeMillis() - start;
            elapsedTimeSec = elapsedTimeMillis / 1000F;
            elapsedTimeSecInMemory = elapsedTimeSec ;
            LOG.info(" Elapsed-Time (sec) inMemory :" + elapsedTimeSec);

            table.flushCommits();
            //disk write elapsed time
            elapsedTimeMillis = System.currentTimeMillis() - start;
            elapsedTimeSec = elapsedTimeMillis / 1000F;
            elapsedTimeSecDiskWrite   = elapsedTimeSec;
            LOG.info(" Elapsed-Time (sec) Disk Write:" +
elapsedTimeSec);
            .........
            .........

My hbase-site.xml is looks like :
 <configuration>
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://master.bigdata.com:54310/hbase</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>master.bigdata.com,slave1.bigdata.com,slave2.bigdata.com,
slave3.bigdata.com</value>
    </property>
    <property>
        <name>hbase.zookeeper.dns.interface</name>
        <value>eth0</value>
    </property>
    <property>
        <name>hbase.zookeeper.dns.nameserver</name>
        <value>10.10.10.1</value>
    </property>

    <property>
        <name>hbase.regionserver.handler.count</name>
        <value>20</value>
    </property>

    <property>
        <name>hbase.client.write.buffer</name>
        <value>5097152</value>
    </property>
</configuration>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message