htrace-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Masatake Iwasaki <iwasak...@oss.nttdata.co.jp>
Subject Re: Trace HBase/HDFS with HTrace
Date Thu, 26 Feb 2015 09:07:42 GMT
 > perhaps a double detach.

It turned out to be double detach of NullScope singleton and I filed 
HTRACE-119.
I got the error on meta assignments on regionserver startup without 
starting trace spans..

Thanks,
Masatake

On 2/26/15 09:58, Colin P. McCabe wrote:
> Hmm.  Looking at that error, my guess would be that there is an
> incorrect usage of TraceScope#detach going on somewhere in hbase...
> perhaps a double detach.  But I could be wrong.  We added some code
> recently to catch issues like this.
>
> best,
> Colin
>
> On Wed, Feb 25, 2015 at 12:28 AM, Masatake Iwasaki
> <iwasakims@oss.nttdata.co.jp> wrote:
>> I tried hbase-1 built against today's htrace-3.2.0-SNAPSHOT (with quick fix
>> to TestHTraceHooks).
>> I got the error below in regionserver log.
>> I will dig this tomorrow.::
>>
>>    2015-02-25 00:18:29,270 ERROR [RS_OPEN_META-centos7:16201-0]
>> htrace.Tracer: Tried to detach trace span null but it has already been
>> detached.
>>    2015-02-25 00:18:29,271 ERROR [RS_OPEN_META-centos7:16201-0]
>> handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740,
>> starting to roll back the global memstore size.
>>    java.lang.RuntimeException: Tried to detach trace span null but it has
>> already been detached.
>>            at org.apache.htrace.Tracer.clientError(Tracer.java:61)
>>            at org.apache.htrace.TraceScope.detach(TraceScope.java:57)
>>            at
>> org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1559)
>>            at
>> org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:94)
>>            at
>> org.apache.hadoop.hbase.regionserver.HRegion.writeRegionOpenMarker(HRegion.java:910)
>>            at
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4911)
>>            at
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4874)
>>            at
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4845)
>>            at
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4801)
>>            at
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4752)
>>            at
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
>>            at
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
>>            at
>> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>>            at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>            at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>            at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> On 2/24/15 18:27, Colin P. McCabe wrote:
>>> Thanks for trying this, Mastake.  I've got HDFS working on my cluster
>>> with tracing and LocalFileSpanReceiver.  Did you try using HBase +
>>> HDFS with LocalFileSpanReceiver?  Be sure to use a build including
>>> HTRACE-112 since LFSR was kind of busted prior to that.
>>>
>>> I'm going to do a longer writeup about getting HDFS + HBase working
>>> with other span receivers just as soon as I finish stomping a few more
>>> bugs.
>>>
>>> best,
>>> Colin
>>>
>>> On Tue, Feb 24, 2015 at 12:04 PM, Masatake Iwasaki
>>> <iwasakims@oss.nttdata.co.jp> wrote:
>>>> Hi,
>>>>
>>>> Thanks for trying this. I am sorry for late reply.
>>>>
>>>> I tried this today
>>>> by hbase-1.0.1-SANPSHOT built with
>>>> {{-Dhadoop-two.version=2.7.0-SNAPSHOT}}
>>>> in pseudo distributed cluster
>>>> but failed to get end-to-end trace.
>>>>
>>>> I checked that
>>>> * tracing works for both of hbase and hdfs,
>>>> * hbase runs with 2.7.0-SNAPSHOT jar of hadoop.
>>>>
>>>> When I did do put with tracing on,
>>>> I saw span named "FSHLog.sync" with annotations such as
>>>> "syncing writer" and "writer synced".
>>>> The code for tracing in FSHLog worked at least.
>>>>
>>>> I'm still looking into this.
>>>> If it turned out that tracing spans are not reached to
>>>> actual HDFS writer thread in HBase, I will file a JIRA.
>>>>
>>>> # We need hadoop-2.6.0 or higher in order to trace HDFS.
>>>> # Building hbase from source with {{-Dhadoop-two.version=2.6.0}}
>>>> # is straight forward way to do this
>>>> # because the binary release of hbase-1.0.0 bundles hadoop-2.5.1 jars.
>>>>
>>>> Masatake
>>>>
>>>>
>>>> On 2/11/15 08:56, Nick Dimiduk wrote:
>>>>> Hi Joshua,
>>>>>
>>>>> In theory there's nothing special for you to do. Just issue your query
>>>>> to
>>>>> HBase with tracing enabled. The active span will go through HBase, down
>>>>> into HDFS, and back again. You'll need both systems collecting spans
>>>>> into
>>>>> the same place so that you can report on the complete trace tree.
>>>>>
>>>>> I've not recently tested the end-to-end, but I believe it's all there.
>>>>> If
>>>>> not, it's a bug -- this is an intended use case. Can you give it a try
>>>>> and let us know how it goes?
>>>>>
>>>>> FYI, 0.99.x are preview releases of HBase and not for production use.
>>>>> Just
>>>>> so you know :)
>>>>>
>>>>> -n
>>>>>
>>>>> On Wednesday, February 11, 2015, Chunxu Tang <chunxutang@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Now I’m exploiting HTrace to trace request level data flows in
HBase
>>>>>> and
>>>>>> HDFS. I have successfully traced HBase and HDFS by using HTrace,
>>>>>> respectively.
>>>>>>
>>>>>> After that, I combine HBase and HDFS together and I want to just
send a
>>>>>> PUT/GET request to HBase, but to trace the whole data flow in both
>>>>>> HBase
>>>>>> and HDFS. In my opinion, when I send a request such as Get to HBase,
it
>>>>>> will at last try to read the blocks on HDFS, so I can construct a
whole
>>>>>> data flow tracing through HBase and HDFS. While, the fact is that
I can
>>>>>> only get tracing data of HBase, with no data of HDFS.
>>>>>>
>>>>>> Could you give me any suggestions on how to trace the data flow in
both
>>>>>> HBase and HDFS? Does anyone have similar experience? Do I need to
>>>>>> modify
>>>>>> the source code? And maybe which part(s) should I touch? If I need
to
>>>>>> modify the code, I will try to create a patch for that.
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>> My Configurations:
>>>>>> Hadoop version: 2.6.0
>>>>>> HBase version: 0.99.2
>>>>>> HTrace version: htrace-master
>>>>>> OS: Ubuntu 12.04
>>>>>>
>>>>>>
>>>>>> Joshua
>>>>>>


Mime
View raw message