hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4486) Add log category for long-running DFSClient notices
Date Thu, 14 Aug 2014 18:26:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14097332#comment-14097332
] 

Colin Patrick McCabe commented on HDFS-4486:
--------------------------------------------

[~oldcap]: there are a bunch of conditions in the DFSClient that are not fatal, but which
degrade performance.  For example, lacking libhadoop.so means you won't be able to use short-circuit.
 Perhaps the short-circuit socket stuff is not set up correctly.  Or we have to fall back
to the shell-based groups plugin.  I believe those are the cases we'd start with here.  The
idea is to have one log setting you could change to see all the "falling back to lower performance
method" logs.

> Add log category for long-running DFSClient notices
> ---------------------------------------------------
>
>                 Key: HDFS-4486
>                 URL: https://issues.apache.org/jira/browse/HDFS-4486
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Todd Lipcon
>            Assignee: Zhe Zhang
>            Priority: Minor
>
> There are a number of features in the DFS client which are transparent but can make a
fairly big difference for performance -- two in particular are short circuit reads and native
checksumming. Because we don't want log spew for clients like "hadoop fs -cat" we currently
log only at DEBUG level when these features are disabled. This makes it difficult to troubleshoot/verify
for long-running perf-sensitive clients like HBase.
> One simple solution is to add a new log category - eg o.a.h.h.DFSClient.PerformanceAdvisory
- which long-running clients could enable at DEBUG level without getting the full debug spew.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message