hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4486) Add log category for long-running DFSClient notices
Date Fri, 22 Aug 2014 20:12:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107409#comment-14107409

Colin Patrick McCabe commented on HDFS-4486:

It's pretty strange, all right.  I'm also seeing a lot of "too many open files" exceptions,
which makes me wonder if the JVM is failing to load the jar file because it's out of file

org/apache/hadoop/util/PerformanceAdvisory.java is certainly in hadoop-common, so there should
be no problem depending on it in hadoop-hdfs.  The hadoop-common unit tests also pass prior
to the hadoop-hdfs tests throwing this exception, so we know that the class got compiled.

Just a random guess, but maybe try adding a reference to PerformanceAdvisory inside MiniDFSCluster.java,
to force Maven to realize that there is a dependency?  Or try adding a constructor or other
method to PerformanceAdvisory.. maybe it's getting somehow optimized out since it only has
a static member now?

> Add log category for long-running DFSClient notices
> ---------------------------------------------------
>                 Key: HDFS-4486
>                 URL: https://issues.apache.org/jira/browse/HDFS-4486
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Todd Lipcon
>            Assignee: Zhe Zhang
>            Priority: Minor
>         Attachments: HDFS-4486-20140820.patch, HDFS-4486-20140821.patch, HDFS-4486-20140822.patch,
> There are a number of features in the DFS client which are transparent but can make a
fairly big difference for performance -- two in particular are short circuit reads and native
checksumming. Because we don't want log spew for clients like "hadoop fs -cat" we currently
log only at DEBUG level when these features are disabled. This makes it difficult to troubleshoot/verify
for long-running perf-sensitive clients like HBase.
> One simple solution is to add a new log category - eg o.a.h.h.DFSClient.PerformanceAdvisory
- which long-running clients could enable at DEBUG level without getting the full debug spew.

This message was sent by Atlassian JIRA

View raw message