hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tahseen Jamal <tahseen.ja...@gmail.com>
Subject Re: In regard to HBase concurrent request
Date Wed, 21 May 2014 03:37:18 GMT

Data retrieved is pretty small. Almost in bytes. It is basically user profile. Might not be
more than 256 bytes data per row 

The configurations you are telling is mostly used in map reduce

The problem that we are facing is that the HBase thread pool is not working. Below is the
code for the same.


===========================================================================================================
 
import java.io.IOException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;

public class TestPool{
	
	private static final byte[] tableName ="sh_self_profiles".getBytes();
	private static Configuration conf;
	private static HTable table;
	static{
		conf = HBaseConfiguration.create();
		 conf.set("hbase.zookeeper.quorum", "10.64.0.156");
	        conf.set("hbase.zookeeper.property.clientPort","2181");
	        conf.set("hbase.master.port", "60000");
	        conf.setInt("hbase.htable.threads.max", 1000);
	        try {
				table=new HTable(conf, tableName);
			} catch (IOException e) {
				e.printStackTrace();
			}
	      
	       
	}
	
	
	public static void main(String[] args) {
		//Here we are generating and hitting concurrent request
		int nbThreads = 100;
		ExecutorService execService = Executors.newFixedThreadPool(nbThreads);
		for (int i = 0; i < nbThreads; i++) {
			
			execService.execute(new Runnable(){
				
				public void run(){
					try {
						String caller="9197100"+(int)(Math.random()*100000);
						if(caller.length()==12){
							TestPool.getOneRecord("sh_self_profiles",caller);
						}
					} catch (Throwable e) {
						e.printStackTrace();
					}
					
				}//end-run
			
			}//end-runnable

			);//end-submit

		}//end-for
		
		

	}

	//this method is used for getting result from hbase
	 public static void getOneRecord (String tableName, String rowKey) throws IOException{
		 try{
			 long startTime=System.currentTimeMillis();
			
			 Get get = new Get(Bytes.toBytes(rowKey));
			 
		        Result rs = table.get(get);
		        long endTime=System.currentTimeMillis();
		        System.out.println("total time "+(endTime-startTime));
		        System.out.println(Integer.parseInt(new String(rs.getValue(Bytes.toBytes("sprofile"),
Bytes.toBytes("male")))));
		        
	        }catch (Exception e) {
				e.printStackTrace();
			}
	    }
	
	 
	
	
}

========================================================================================================================


On 21-May-2014, at 8:56 am, Mikhail Antonov <olorinbant@gmail.com> wrote:

> How much data did you retrieve with single Get? if it's pretty small, you
> could try increasing this param.
> 
> See 2.5.2.3 in http://hbase.apache.org/book/important_configurations.html
> 
> 
> 2014-05-20 20:21 GMT-07:00 Tahseen Jamal <tahseen.jamal@gmail.com>:
> 
>> Yes, have use the method you mentioned
>> 
>> And the value of hbase.regionserver.handler.count is around 40
>> 
>> 
>> 
>> 
>> On 21-May-2014, at 8:38 am, Ted Yu <yuzhihong@gmail.com> wrote:
>> 
>>> Did you use the following method from HTable ?
>>> 
>>> public Result[] get(List<Get> gets) throws IOException {
>>> 
>>> What is the value for hbase.regionserver.handler.count ?
>>> 
>>> Cheers
>>> 
>>> 
>>> On Tue, May 20, 2014 at 8:03 PM, Tahseen Jamal <tahseen.jamal@gmail.com
>>> wrote:
>>> 
>>>> Dear All,
>>>> 
>>>> 
>>>> I have 10 million records in a table. when i get single column value
>> from
>>>> hbase it takes around 10ms. My problem is when I hit 100 or more
>> concurrent
>>>> request the time slowly accumulates and increases to more than 400 ms
>>>> instead of completing in 10ms only. When 100 requests are hit linearly
>> each
>>>> one takes 10ms only. I am using hbase-0.96 version.
>>>> 
>>>> 
>> 
>> 
> 
> 
> -- 
> Thanks,
> Michael Antonov


Mime
View raw message