htrace-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Masatake Iwasaki <iwasak...@oss.nttdata.co.jp>
Subject Re: Trace HBase/HDFS with HTrace
Date Wed, 25 Feb 2015 08:28:44 GMT
I tried hbase-1 built against today's htrace-3.2.0-SNAPSHOT (with quick 
fix to TestHTraceHooks).
I got the error below in regionserver log.
I will dig this tomorrow.::

   2015-02-25 00:18:29,270 ERROR [RS_OPEN_META-centos7:16201-0] 
htrace.Tracer: Tried to detach trace span null but it has already been 
detached.
   2015-02-25 00:18:29,271 ERROR [RS_OPEN_META-centos7:16201-0] 
handler.OpenRegionHandler: Failed open of 
region=hbase:meta,,1.1588230740, starting to roll back the global 
memstore size.
   java.lang.RuntimeException: Tried to detach trace span null but it 
has already been detached.
           at org.apache.htrace.Tracer.clientError(Tracer.java:61)
           at org.apache.htrace.TraceScope.detach(TraceScope.java:57)
           at 
org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1559)
           at 
org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:94)
           at 
org.apache.hadoop.hbase.regionserver.HRegion.writeRegionOpenMarker(HRegion.java:910)
           at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4911)
           at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4874)
           at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4845)
           at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4801)
           at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4752)
           at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
           at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
           at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
           at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
           at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
           at java.lang.Thread.run(Thread.java:745)


On 2/24/15 18:27, Colin P. McCabe wrote:
> Thanks for trying this, Mastake.  I've got HDFS working on my cluster
> with tracing and LocalFileSpanReceiver.  Did you try using HBase +
> HDFS with LocalFileSpanReceiver?  Be sure to use a build including
> HTRACE-112 since LFSR was kind of busted prior to that.
>
> I'm going to do a longer writeup about getting HDFS + HBase working
> with other span receivers just as soon as I finish stomping a few more
> bugs.
>
> best,
> Colin
>
> On Tue, Feb 24, 2015 at 12:04 PM, Masatake Iwasaki
> <iwasakims@oss.nttdata.co.jp> wrote:
>> Hi,
>>
>> Thanks for trying this. I am sorry for late reply.
>>
>> I tried this today
>> by hbase-1.0.1-SANPSHOT built with {{-Dhadoop-two.version=2.7.0-SNAPSHOT}}
>> in pseudo distributed cluster
>> but failed to get end-to-end trace.
>>
>> I checked that
>> * tracing works for both of hbase and hdfs,
>> * hbase runs with 2.7.0-SNAPSHOT jar of hadoop.
>>
>> When I did do put with tracing on,
>> I saw span named "FSHLog.sync" with annotations such as
>> "syncing writer" and "writer synced".
>> The code for tracing in FSHLog worked at least.
>>
>> I'm still looking into this.
>> If it turned out that tracing spans are not reached to
>> actual HDFS writer thread in HBase, I will file a JIRA.
>>
>> # We need hadoop-2.6.0 or higher in order to trace HDFS.
>> # Building hbase from source with {{-Dhadoop-two.version=2.6.0}}
>> # is straight forward way to do this
>> # because the binary release of hbase-1.0.0 bundles hadoop-2.5.1 jars.
>>
>> Masatake
>>
>>
>> On 2/11/15 08:56, Nick Dimiduk wrote:
>>> Hi Joshua,
>>>
>>> In theory there's nothing special for you to do. Just issue your query to
>>> HBase with tracing enabled. The active span will go through HBase, down
>>> into HDFS, and back again. You'll need both systems collecting spans into
>>> the same place so that you can report on the complete trace tree.
>>>
>>> I've not recently tested the end-to-end, but I believe it's all there. If
>>> not, it's a bug -- this is an intended use case. Can you give it a try
>>> and let us know how it goes?
>>>
>>> FYI, 0.99.x are preview releases of HBase and not for production use. Just
>>> so you know :)
>>>
>>> -n
>>>
>>> On Wednesday, February 11, 2015, Chunxu Tang <chunxutang@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Now I’m exploiting HTrace to trace request level data flows in HBase and
>>>> HDFS. I have successfully traced HBase and HDFS by using HTrace,
>>>> respectively.
>>>>
>>>> After that, I combine HBase and HDFS together and I want to just send a
>>>> PUT/GET request to HBase, but to trace the whole data flow in both HBase
>>>> and HDFS. In my opinion, when I send a request such as Get to HBase, it
>>>> will at last try to read the blocks on HDFS, so I can construct a whole
>>>> data flow tracing through HBase and HDFS. While, the fact is that I can
>>>> only get tracing data of HBase, with no data of HDFS.
>>>>
>>>> Could you give me any suggestions on how to trace the data flow in both
>>>> HBase and HDFS? Does anyone have similar experience? Do I need to modify
>>>> the source code? And maybe which part(s) should I touch? If I need to
>>>> modify the code, I will try to create a patch for that.
>>>>
>>>> Thank you.
>>>>
>>>> My Configurations:
>>>> Hadoop version: 2.6.0
>>>> HBase version: 0.99.2
>>>> HTrace version: htrace-master
>>>> OS: Ubuntu 12.04
>>>>
>>>>
>>>> Joshua
>>>>


Mime
View raw message