hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sushanth Sowmyan (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-14394) Reduce excessive INFO level logging
Date Sun, 31 Jul 2016 19:19:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401311#comment-15401311
] 

Sushanth Sowmyan edited comment on HIVE-14394 at 7/31/16 7:18 PM:
------------------------------------------------------------------

The recent event that predicated the filing of this bug is reports that we're filling up disks
of the metastore box from recent hive installations, due to the excessive amount of metrics
logging we generate. At first, this is not a hive problem, because the primary bit of extra
logging is coming from HadoopMetrics2Reporter, which is a new library we depend on to publish
our Codahale metrics to the Hadoop Metrics2 subsystem, and is easily fixed by changing the
loglevel for that class to DEBUG, rather than retaining it on INFO. However, it becomes our
problem for upgrade scenarios where the customer now has a manual change they need to make
for smooth operations of their cluster because we upgraded a dependency.

Thus, we need to work with the maintainer ([~elserj]) of HadoopMetrics2Reporter to get a new
version of the library, with info-level logging reduced to debug-level logging. To this end,
I have filed a github pull request : https://github.com/joshelser/dropwizard-hadoop-metrics2/pull/1
. 

However, in the meanwhile, we also need to make a patch available (even if we don't commit
it to hive) to solve the problem those installations have. Thus, I'm uploading a temporary
patch to move the Metrics2 Reporter to the hive project, with the fix, so that those that
need it can use this patch. I will update this patch with a newer patch which bumps up the
version number of the dependency with the log-level aspect changed once we have a newer library
version with that.


was (Author: sushanth):
The recent event that predicated the filing of this bug is reports that we're filling up disks
of the metastore box from recent hive installations, due to the excessive amount of metrics
logging we generate. At first, this is not a hive problem, because the primary bit of extra
logging is coming from HadoopMetrics2Reporter, which is a new library we depend on to publish
our Codahale metrics to the Hadoop Metrics2 subsystem, and is easily fixed by changing the
loglevel for that class to DEBUG, rather than retaining it on INFO. However, it becomes our
problem for upgrade scenarios where the customer now has a manual change they need to make
for smooth operations of their cluster because we upgraded a dependency.

Thus, we need to work with the maintainer ([~elserj] of HadoopMetrics2Reporter to get a new
version of the library, with info-level logging reduced to debug-level logging. To this end,
I have filed a github pull request : https://github.com/joshelser/dropwizard-hadoop-metrics2/pull/1
. 

However, in the meanwhile, we also need to make a patch available (even if we don't commit
it to hive) to solve the problem those installations have. Thus, I'm uploading a temporary
patch to move the Metrics2 Reporter to the hive project, with the fix, so that those that
need it can use this patch. I will update this patch with a newer patch which bumps up the
version number of the dependency with the log-level aspect changed once we have a newer library
version with that.

> Reduce excessive INFO level logging
> -----------------------------------
>
>                 Key: HIVE-14394
>                 URL: https://issues.apache.org/jira/browse/HIVE-14394
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>
> We need to cull down on the number of logs we generate in HMS and HS2 that are not needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message