hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arpit Agarwal <aagar...@hortonworks.com>
Subject Re: Questions on HDFS-8880
Date Tue, 06 Oct 2015 20:11:47 GMT
There is no property called hadoop.metrics.log.file. Where do you see it? And there is no property
that would show up in the command-line of every process. namenode.metrics.logger would show
up in the NN command-line and even that is off by default.

FileSink looks limited for practical use.
- No support for rolling output files.
- The format is hard-coded and not practical to grep. Changing it is incompatible.
- Won't write to new output files unless I create the files first with 'touch'.

- Not all NN metrics are logged. These last two are likely bugs.

On 10/6/15, 11:46 AM, "Allen Wittenauer" <aw@altiscale.com> wrote:

>Current state is the custom stuff in HDFS-8880 is mostly undocumented except for some
tuning bits in hdfs-default.xml vs. the javadocs for metrics2.  Neither of which is ideal.
>There’s no doubt the documentation for all of the metrics2 sinks are… sparse.  Which
is likely what led to the duplicate functionality. :( 
>On Oct 6, 2015, at 11:29 AM, Andrew Wang <andrew.wang@cloudera.com> wrote:
>> If it's duplicate we should probably back it out, but taking a step back,
>> is the issue that there isn't good documentation about configuring Metrics2
>> / FileSync? I see the API docs, but a user-focused guide on how to
>> configure Metrics2 would probably be a welcome addition.
>> HBase has a blog at https://blogs.apache.org/hbase/ this could also be good
>> content for a blog post.
>> Best,
>> Andrew
>> On Tue, Oct 6, 2015 at 11:12 AM, Allen Wittenauer <aw@altiscale.com> wrote:
>>> Folks,
>>>        I’ve been looking over HDFS-8880 and it’s various follow-on
>>> JIRAs.  The intentions are good, but the implementation is
>>> mostly/effectively a duplicate of the FileSink that’s already part of the
>>> Hadoop metrics subsystem. (which therefore means it works with all daemons,
>>> out of the box already).  Reading through HDFS-9114, it’s pretty obvious
>>> now that users are going to get *very* confused as just what happens when
>>> they set the “hadoop.metrics.log.file” property.  It’s opening a pandora’s
>>> box of work, since that property only partially works with one sub-project,
>>> will show up on the command line of every daemon, and isn’t documented...
>>>        I’d like to see this series of patches reverted (they haven’t
>>> shipped yet, so now is the time!) and effort placed into updating the
>>> metrics2 FileSink to have whatever functionality is missing.
>>>        Thoughts?
View raw message