hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6881) The DFSClient should use the sampler to determine whether to initiate trace spans when making RPCv9 calls to the NN and DN
Date Mon, 22 Sep 2014 06:28:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142940#comment-14142940
] 

Colin Patrick McCabe commented on HDFS-6881:
--------------------------------------------

There are a lot of DistributedFileSystem operations that just involve making an RPC to the
NameNode or the DataNode via RPCv9.  We currently never trace those, unless there is already
an active trace span.  This patch changes this so that RPCv9 calls will use the HTrace {{Sampler}}
configured by the DFSClient to decide whether to initiate a trace span when making an RPC.

This is done by adding a new method to the {{RpcInvocationHandler}} interface, which is implemented
by {{ProtobufRpcEngine#Invoker}}.  This new method changes the sampler that the invoker is
using.  The {{DFSClient}} uses it to tell the proxy object it creates which HTrace sampler
to use.

There a little bit of overlap between this patch and the patch in HDFS-7055 (Add tracing to
DFSInputStream), since both patches rely on the {{dfs.client.trace.sampler}} option being
present.  But both patches can compile and be committed on their own.

I changed the trace span name format that we're using slightly.  Formerly, we would populate
trace spans with names like {{org.apache.hadoop.hdfs.protocol.ClientProtocol.getServerDefaults}}.
 This is a rather long name, and I think {{ClientProtocol#getServerDefaults}} is just as descriptive,
and unlikely to lead to any confusion.

Requests from HTrace for a configuration key X now cause it to access {{hadoop.htrace.X}}
rather than {{hadoop.X}}.  Hopefully, this will avoid potential configuration key name conflicts
for any possible HTrace configuration key someone might want to implement.  I also modified
the class search to prepend {{org.htrace.impl.}} to class names with no namespace given. 
This should lead to shorter configurations, and also protect us against any changes in HTrace
in the future (like moving the span receivers to a different namespace).

SpanReceiverHost now sets {{hadoop.htrace.local-file-span-receiver.path}} to {{/tmp/\$PID}}
if it is not already set.  This makes it a lot easier to test tracing, since you can just
feed all daemons the same config and get nice trace files in {[/tmp}}.

> The DFSClient should use the sampler to determine whether to initiate trace spans when
making RPCv9 calls to the NN and DN
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-6881
>                 URL: https://issues.apache.org/jira/browse/HDFS-6881
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>    Affects Versions: 2.6.0
>            Reporter: Masatake Iwasaki
>            Assignee: Colin Patrick McCabe
>
> The DFSClient should use the configred HTrace sampler to determine whether to initiate
trace spans when making RPCv9 calls to the NN and DN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message