flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ufuk Celebi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-5207) Decrease HadoopFileSystem logging
Date Tue, 29 Nov 2016 16:02:59 GMT
Ufuk Celebi created FLINK-5207:

             Summary: Decrease HadoopFileSystem logging
                 Key: FLINK-5207
                 URL: https://issues.apache.org/jira/browse/FLINK-5207
             Project: Flink
          Issue Type: Improvement
            Reporter: Ufuk Celebi

HadoopFileSystem logging is very noisy when trying to get the Hadoop configuration, which
happens for every HadoopFileSystem. This happens for example when a checkpoint is written
to HDFS in case of HA.

1) Two log messages relate to the user config:

final String hdfsDefaultPath = GlobalConfiguration.getString(ConfigConstants.HDFS_DEFAULT_CONFIG,
		if (hdfsDefaultPath != null) {
			retConf.addResource(new org.apache.hadoop.fs.Path(hdfsDefaultPath));
		} else {
			LOG.debug("Cannot find hdfs-default configuration file");

Since this is reflected in the logged configuration properties, it is mostly unnecessary to
log over and over again. Therefore I propose to go with TRACE.

(2) Two log messages relate to the default config files of Hadoop. There I would change the
debug message to happen only when they are *not* found. If it is still too noisy, we could
completely remove it or decrease to TRACE at another time.

This message was sent by Atlassian JIRA

View raw message