hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kireet <kir...@feedly.com>
Subject Re: HConnectionManager$HConnectionImplementation.locateRegionInMeta
Date Fri, 31 May 2013 18:58:02 GMT


Even if I initiate the call via a pooled htable, the MetaScanner seems 
to use a concrete HTable instance. The constructor invoked seems to 
create a java ThreadPoolExecutor. I am not 100% sure but I think as long 
as nothing is submitted to the ThreadPoolExecutor it won't create any 
threads. I just wanted to confirm this was the case. I do see the 
connection is shared.

--Kireet



On 5/30/13 7:38 PM, Ted Yu wrote:
> HTablePool$**PooledHTable is a wrapper around HTable.
>
> Here is how HTable obtains a connection:
>
>    public HTable(Configuration conf, final byte[] tableName, final
> ExecutorService pool)
>        throws IOException {
>      this.connection = HConnectionManager.getConnection(conf);
>
> Meaning the connection is a shared one based on certain key/value pairs
> from conf.
>
> bq. So every call to batch will create a new thread?
>
> I don't think so.
>
> On Thu, May 30, 2013 at 11:28 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+AlfA@public.gmane.org>
wrote:
>
>>
>>
>> Thanks, will give it a shot. So I should download 0.94.7 (latest stable)
>> and run the patch tool on top with the backport? This is a little new to me.
>>
>> Also, I was looking at the stack below. From my reading of the code, the
>> HTable.batch() call will always cause the prefetch call to occur, which
>> will cause a new HTable object to get created. The constructor used in
>> creating a new thread pool. So every call to batch will create a new
>> thread? Or the HTable's thread pool never gets used as the pool is only
>> used for writes? I think I am missing something but just want to confirm.
>>
>> Thanks
>> Kireet
>>
>> On 5/30/13 12:48 PM, Himanshu Vashishtha wrote:
>>
>>> bq. Anoop attached backported patch in HBASE-8655. It should go into
>>>
>>> 0.94.9, the next release - current is 0.94.8
>>>
>>> In case you want it sooner, you can apply 8655 patch and test/verify it.
>>>
>>> Thanks,
>>> Himanshu
>>>
>>>
>>>
>>> On Thu, May 30, 2013 at 7:26 AM, Ted Yu <yuzhihong-**
>>> Re5JQEeQqe8AvxtiuMwx3w@public.**gmane.org<yuzhihong-Re5JQEeQqe8AvxtiuMwx3w-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org>>
>>> wrote:
>>>
>>>   Anoop attached backported patch in HBASE-8655
>>>>
>>>> It should go into 0.94.9, the next release - current is 0.94.8
>>>>
>>>> Cheers
>>>>
>>>> On Thu, May 30, 2013 at 7:01 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
>>>> AlfA-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org <kireet-Teh5dPVPL8nQT0dZR%2BAlfA-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org>>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> How long do backports typically take? We have to go live in a month
>>>>> ready
>>>>> or not. Thanks for the quick replies Anoop and Ted.
>>>>>
>>>>> --Kireet
>>>>>
>>>>>
>>>>> On 5/30/13 9:20 AM, Ted Yu wrote:
>>>>>
>>>>>   0.95 client is not compatible with 0.94 cluster. So you cannot use
0.95
>>>>>> client.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On May 30, 2013, at 6:12 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
>>>>>> AlfA-XMD5yJDbdMReXY1tMh2IBg@**public.gmane.org<AlfA-XMD5yJDbdMReXY1tMh2IBg-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org><
>>>>>> kireet-Teh5dPVPL8nQT0dZR%**2BAlfA-XMD5yJDbdMReXY1tMh2IBg@**
>>>>>> public.gmane.org<kireet-Teh5dPVPL8nQT0dZR%252BAlfA-XMD5yJDbdMReXY1tMh2IBg-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org>
>>>>>>
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Would there be a problem if our cluster is 0.94 and we use a
0.95
>>>>>>>
>>>>>> client?
>>>>
>>>>>
>>>>>>> I am not familiar with the HBase code base, but I did a dump
of the
>>>>>>> thread that is actually running (below). It seems like it is
related
>>>>>>>
>>>>>> to the
>>>>
>>>>> issue you mentioned as the running thread is doing the prefetch logic.
>>>>>>> Would pre-splitting tables help here? We are doing some performance
>>>>>>>
>>>>>> tests
>>>>
>>>>> and essentially starting from an empty instance.
>>>>>>>
>>>>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>>>>> at java.lang.Object.wait(Native Method)
>>>>>>> at java.lang.Object.wait(Object.****java:503)
>>>>>>> at org.apache.zookeeper.****ClientCnxn.submitRequest(**
>>>>>>> ClientCnxn.java:1309)
>>>>>>> - locked <0x00000000e10cf830> (a org.apache.zookeeper.**
>>>>>>> ClientCnxn$Packet)
>>>>>>> at org.apache.zookeeper.****ZooKeeper.exists(ZooKeeper.****java:1036)
>>>>>>> at org.apache.hadoop.hbase.****zookeeper.****
>>>>>>> RecoverableZooKeeper.exists(**
>>>>>>> RecoverableZooKeeper.java:172)
>>>>>>> at org.apache.hadoop.hbase.****zookeeper.ZKUtil.checkExists(****
>>>>>>> ZKUtil.java:450)
>>>>>>> at org.apache.hadoop.hbase.****zookeeper.****ZooKeeperNodeTracker.**
>>>>>>> checkIfBaseNodeAvailable(****ZooKeeperNodeTracker.java:208)
>>>>>>> at org.apache.hadoop.hbase.****zookeeper.RootRegionTracker.**
>>>>>>> waitRootRegionLocation(****RootRegionTracker.java:77)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegion(****
>>>>>>> HConnectionManager.java:874)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegionInMeta(**
>>>>>>> HConnectionManager.java:987)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegion(****
>>>>>>> HConnectionManager.java:885)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegion(****
>>>>>>> HConnectionManager.java:846)
>>>>>>> at org.apache.hadoop.hbase.****client.HTable.finishSetup(**
>>>>>>> HTable.java:234)
>>>>>>> at org.apache.hadoop.hbase.****client.HTable.<init>(HTable.****
>>>>>>> java:174)
>>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner.metaScan(**
>>>>>>> MetaScanner.java:160)
>>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner.access$000(****
>>>>>>> MetaScanner.java:54)
>>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner$1.connect(****
>>>>>>> MetaScanner.java:133)
>>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner$1.connect(****
>>>>>>> MetaScanner.java:130)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager.**
>>>>>>> execute(HConnectionManager.****java:383)
>>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner.metaScan(**
>>>>>>> MetaScanner.java:130)
>>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner.metaScan(**
>>>>>>> MetaScanner.java:105)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****prefetchRegionCache(**
>>>>>>> HConnectionManager.java:947)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegionInMeta(**
>>>>>>>
>>>>>>> HConnectionManager.java:1002)
>>>>>>> - locked <0x00000000e882f778> (a java.lang.Object)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegion(****
>>>>>>> HConnectionManager.java:889)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****locateRegion(****
>>>>>>> HConnectionManager.java:846)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>> HConnectionImplementation.****processBatchCallback(**
>>>>>>> HConnectionManager.java:1533)
>>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$**
>>>>>>>
>>>>>>>   HConnectionImplementation.****processBatch(****
>>>> HConnectionManager.java:1418)
>>>>
>>>>> at org.apache.hadoop.hbase.****client.HTable.batch(HTable.****java:702)
>>>>>>> at org.apache.hadoop.hbase.****client.HTable.get(HTable.java:****671)
>>>>>>> at org.apache.hadoop.hbase.****client.HTablePool$**
>>>>>>> PooledHTable.get(HTablePool.****java:367)
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Kireet
>>>>>>>
>>>>>>> On 5/30/13 12:16 AM, Anoop John wrote:
>>>>>>>
>>>>>>>   Can you have a look at issue HBASE-8476?  Seems related?  A
fix is
>>>>>>>> available in HBASE-8346's patch..
>>>>>>>>
>>>>>>>> -Anoop-
>>>>>>>>
>>>>>>>> On Thu, May 30, 2013 at 9:21 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
>>>>>>>> AlfA-XMD5yJDbdMReXY1tMh2IBg@**public.gmane.org<AlfA-XMD5yJDbdMReXY1tMh2IBg-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org>
>>>>>>>> <
>>>>>>>>
>>>>>>> kireet-Teh5dPVPL8nQT0dZR%**2BAlfA-XMD5yJDbdMReXY1tMh2IBg@**
>>>> public.gmane.org<kireet-Teh5dPVPL8nQT0dZR%252BAlfA-XMD5yJDbdMReXY1tMh2IBg-XMD5yJDbdMReXY1tMh2IBg@public.gmane.org>
>>>>>>
>>>>
>>>>   wrote:
>>>>>>>>
>>>>>>>>    We are running hbase 0.94.6 in a concurrent environment
and we are
>>>>>>>>
>>>>>>>>> seeing
>>>>>>>>> the majority of our code stuck in this method at the
synchronized
>>>>>>>>>
>>>>>>>> block
>>>>
>>>>> (line 997). Is there some configuration we can change to avoid it? It
>>>>>>>>> seems
>>>>>>>>> like we could disable prefetching, but that sounds like
it could
>>>>>>>>>
>>>>>>>> cause
>>>>
>>>>> other problems?
>>>>>>>>>
>>>>>>>>>     - org.apache.hadoop.hbase.******client.HConnectionManager$**
>>>>>>>>> HConnectionImplementation.******locateRegionInMeta(byte[],
byte[],
>>>>>>>>>
>>>>>>>>> byte[],
>>>>>>>>> boolean, java.lang.Object, boolean) @bci=149, line=997
(Compiled
>>>>>>>>>
>>>>>>>> frame)
>>>>
>>>>>     - org.apache.hadoop.hbase.******client.HConnectionManager$**
>>>>>>>>> HConnectionImplementation.******locateRegion(byte[],
byte[],
>>>>>>>>> boolean,
>>>>>>>>>
>>>>>>>>> boolean) @bci=212, line=889 (Compiled frame)
>>>>>>>>>     - org.apache.hadoop.hbase.******client.HConnectionManager$**
>>>>>>>>> HConnectionImplementation.******locateRegion(byte[],
byte[])
>>>>>>>>> @bci=5,
>>>>>>>>> line=846
>>>>>>>>> (Compiled frame)
>>>>>>>>>     - org.apache.hadoop.hbase.******client.HConnectionManager$**
>>>>>>>>>
>>>>>>>>>   HConnectionImplementation.******processBatchCallback(java.******
>>>> util.List,
>>>>
>>>>> byte[], java.util.concurrent.******ExecutorService, java.lang.Object[],
>>>>>>>>> org.apache.hadoop.hbase.******client.coprocessor.Batch$******
>>>>>>>>> Callback)
>>>>>>>>> @bci=194,
>>>>>>>>> line=1533 (Compiled frame)
>>>>>>>>>     - org.apache.hadoop.hbase.******client.HConnectionManager$**
>>>>>>>>> HConnectionImplementation.******processBatch(java.util.List,
>>>>>>>>> byte[],
>>>>>>>>> java.util.concurrent.******ExecutorService, java.lang.Object[])
>>>>>>>>>
>>>>>>>> @bci=30,
>>>>
>>>>> line=1418 (Compiled frame)
>>>>>>>>>     - org.apache.hadoop.hbase.******client.HTable.batch(java.util.**
>>>>>>>>> **
>>>>>>>>>
>>>>>>>>> **List)
>>>>>>>>> @bci=24, line=702 (Compiled frame)
>>>>>>>>>     - org.apache.hadoop.hbase.******client.HTable.get(java.util.****
>>>>>>>>> **List)
>>>>>>>>> @bci=2,
>>>>>>>>> line=671 (Compiled frame)
>>>>>>>>>     - org.apache.hadoop.hbase.******client.HTablePool$**
>>>>>>>>> PooledHTable.get(java.util.******List) @bci=5, line=367
(Compiled
>>>>>>>>>
>>>>>>>> frame)
>>>>
>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>


Mime
View raw message