hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject Re: hbase insert performance test (from hbasemaster and regionservers)
Date Mon, 21 May 2012 22:48:59 GMT
I would love to, but day job is still keeping me busy. :-)

I think you can google up the other threads on this... Andrew wrote up a bit more.

Sorry, 

-Mike

On May 21, 2012, at 5:44 PM, Tom Brown wrote:

> Micheal,
> 
> This is good info. I wish you'd post what the "more" is, though.
> 
> --Tom
> 
> On Mon, May 21, 2012 at 4:30 PM, Michael Segel
> <michael_segel@hotmail.com> wrote:
>> Hi,
>> 
>> Seems we just had someone talk about this just the other day...
>> 
>> 1) 8GB of memory isn't enough to run both M/R and HBase.
>> Ok, yes you can run it, however don't expect it to perform well.
>> 
>> 2) You never want a user to run their own code from the cluster itself. Use an *edge*
node.
>> 
>> There's more, but you get the idea.
>> 
>> On May 21, 2012, at 2:04 PM, Faruk Berksöz wrote:
>> 
>>> Dear All,
>>> we have 4 node in our cluster (1nn+dn,3 dn).
>>> Hadoop dist. is cdh3u3.
>>> Every node has 2 tb disk , 8 gb memory.
>>> We are trying some insert performance test on hbase.
>>> I have tried to insert 250.000 records from hbase master  (without thread),
>>> that takes  5-7 sec.
>>> But when I try ro insert from  any regionserver the same data (250.000)  ,
>>> it takes longer  21 sec.
>>> Is that normal ?
>>> 
>>> any response would be appreciated..
>>> 
>>> 
>>> my java code looks like:
>>>            .........
>>>            .........
>>>            .........
>>>            long start = System.currentTimeMillis();
>>>            //DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd
>>> HH:mm:ss");
>>>            //Date sdate = new Date();
>>> 
>>>            //LOG.info(" Start-Time :" + dateFormat.format(sdate));
>>>            for (int i = 0; i < eachsize; i++) {
>>> 
>>> 
>>>                put = new Put(String.format("%016x",
>>> random.nextLong()).getBytes());
>>> 
>>>                put.setWriteToWAL(false);
>>>                //set column and their values
>>>                addColumnAndValues();
>>>                table.put(put);
>>> 
>>> 
>>>            }
>>>            elapsedTimeMillis = System.currentTimeMillis() - start;
>>>            elapsedTimeSec = elapsedTimeMillis / 1000F;
>>>            elapsedTimeSecInMemory = elapsedTimeSec ;
>>>            LOG.info(" Elapsed-Time (sec) inMemory :" + elapsedTimeSec);
>>> 
>>>            table.flushCommits();
>>>            //disk write elapsed time
>>>            elapsedTimeMillis = System.currentTimeMillis() - start;
>>>            elapsedTimeSec = elapsedTimeMillis / 1000F;
>>>            elapsedTimeSecDiskWrite   = elapsedTimeSec;
>>>            LOG.info(" Elapsed-Time (sec) Disk Write:" +
>>> elapsedTimeSec);
>>>            .........
>>>            .........
>>> 
>>> My hbase-site.xml is looks like :
>>> <configuration>
>>>    <property>
>>>        <name>hbase.rootdir</name>
>>>        <value>hdfs://master.bigdata.com:54310/hbase</value>
>>>    </property>
>>>    <property>
>>>        <name>hbase.cluster.distributed</name>
>>>        <value>true</value>
>>>    </property>
>>>    <property>
>>>        <name>hbase.zookeeper.quorum</name>
>>>        <value>master.bigdata.com,slave1.bigdata.com,slave2.bigdata.com,
>>> slave3.bigdata.com</value>
>>>    </property>
>>>    <property>
>>>        <name>hbase.zookeeper.dns.interface</name>
>>>        <value>eth0</value>
>>>    </property>
>>>    <property>
>>>        <name>hbase.zookeeper.dns.nameserver</name>
>>>        <value>10.10.10.1</value>
>>>    </property>
>>> 
>>>    <property>
>>>        <name>hbase.regionserver.handler.count</name>
>>>        <value>20</value>
>>>    </property>
>>> 
>>>    <property>
>>>        <name>hbase.client.write.buffer</name>
>>>        <value>5097152</value>
>>>    </property>
>>> </configuration>
>> 
> 


Mime
View raw message