hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "eric baldeschwieler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-96) name server should log decisions that affect data: block creation, removal, replication
Date Fri, 31 Mar 2006 00:28:02 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-96?page=comments#action_12372603 ] 

eric baldeschwieler commented on HADOOP-96:
-------------------------------------------

One thing that really helped us was to be able to specify the duration to keep log files.
 So you could configure the system to keep up to N seconds of logs (think one month).  This
way logs don't grow without bound, but you can be confident how much history will be available.
 Also, logs gzip pretty well.  It would be nice to zip closed logs automatically.

Don't know how much of this you can get for free from the existing logs packages.  Should
investigate this.

Are we just logging on the name node, or are data nodes logging all events too.  Seems like
that would be desirable as well.  Using the same mechanisms of course.


> name server should log decisions that affect data: block creation, removal, replication
> ---------------------------------------------------------------------------------------
>
>          Key: HADOOP-96
>          URL: http://issues.apache.org/jira/browse/HADOOP-96
>      Project: Hadoop
>         Type: Improvement
>   Components: dfs
>     Versions: 0.1
>     Reporter: Yoram Arnon
>     Assignee: Hairong Kuang
>     Priority: Critical

>
> currently, there's no way to analyze and debug DFS errors where blocks disapear.
> name server should log its decisions that affect data, including block creation, removal,
replication:
> - block <b> created, assigned to datanodes A, B, ...
> - datanode A dead, block <b> underreplicated(1), replicating to datanode C
> - datanode B dead, block <b> underreplicated(2), replicating to datanode D
> - datanode A alive, block <b> overreplicated, removing from datanode D
> - block <removed> from datanodes C, D, ...
> that will enable me to track down, two weeks later, a block that's missing from a file,
and to debug the name server.
> extra credit:
> - rotate log file, as it might grow large
> - make this behaviour optional/configurable

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message