hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karan Mehta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11583) Parent spans not initialized to NullScope for every DFSPacket
Date Fri, 31 Mar 2017 18:48:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951482#comment-15951482

Karan Mehta commented on HDFS-11583:

Just for FYI,
Current Phoenix uses HBase-1.2.3 and Hadoop - 2.7.1, which all depend on HTrace-3.1
This fact suggests for another possible change. We also need to add {{traceId}} as one the
class variables along with the {{parents}} for each {{DFSPacket}} we receive with tracing
enabled. Hardcoding it to {{0}} doesn't seem appropriate and might as well create confusions
on the Tracer output. Different traces might be perceived as a single scope since their {{traceId}}
is same. Any thoughts on this? Should we create a separate JIRA for this?

The patch seems good. Thanks [~iwasakims].  
Although it is not really an issue, should we also consider doing the same to {{ResponseProcessor}}
(redefine the {{scope}} variable to {{NULLSCOPE.Instance}}? This is more of a safety measure
so that we don't run into troubles if new spans might be added over there in future.

> Parent spans not initialized to NullScope for every DFSPacket
> -------------------------------------------------------------
>                 Key: HDFS-11583
>                 URL: https://issues.apache.org/jira/browse/HDFS-11583
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: tracing
>    Affects Versions: 2.7.1
>            Reporter: Karan Mehta
>            Assignee: Masatake Iwasaki
>         Attachments: HDFS-11583-branch-2.7.001.patch, HDFS-11583-branch-2.7.002.patch
> The issue was found while working with PHOENIX-3752.
> Each packet received by the {{run()}} method of {{DataStreamer}} class, uses the {{parents}}
field of the {{DFSPacket}} to create a new {{dataStreamer}} span, which in turn creates a
{{writeTo}} span as its child span. The parents field is initialized when the packet is added
to the {{dataQueue}} and the value is initialized from the {{ThreadLocal}}. This is how HTrace
handles spans. 
> A {{TraceScope}} is created and initialized to {{NullScope}} before the loop which runs
till the point when the stream is closed. 
> Consider the following scenario, when the {{dataQueue}} contains multiple packets, only
the first of which has a tracing enabled. The scope is initialized to the {{dataStreamer}}
scope and a {{writeTo}} span is created as its child, which gets closed once the packet is
sent out to a remote datanode. Before {{writeTo}} span is started, the {{dataStreamer}} scope
is detached. So calling the close method on it doesn't do anything at the end of loop. 
> The second iteration will be using the stale value of the {{scope}} variable with a DFSPacket
on which tracing is not enabled. This results in generation of an orphan {{writeTo}} spans
which are being delivered to the {{SpanReceiver}} as registered in the TraceFramework. This
may result in unlimited number of spans being generated and sent out to receiver. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message