hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13717) Shell scripts call hadoop_verify_logdir even when command is not started as daemon
Date Fri, 14 Oct 2016 19:49:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576304#comment-15576304

Allen Wittenauer commented on HADOOP-13717:

bq.  it seems like if someone is not specifying the "--daemon" flag, then they don't care
about daemon things like pid files and log dirs for stdout/stderr.

They do, actually.  The inconsistent behavior of the hadoop daemons was a big sticking point
amongst quite a few admins I had talked to.  

bq. The audit log is an interesting case, but I think app-specific logging should be checked
in the app, not the shell scripts (which are generic).

I'd love to see pid and log dir handling out of the scripts.  It greatly over-complicates
them. One thing to keep in mind that doing it prior to Java launch means that we get extremely
fast fail: there's no Java classpath work and no Java initialization costs.

bq.  Bigtop and CDH don't have balancer init scripts for instance.

Sorry, I think I may have miscommunicated this point.  start-balancer is geared towards manual
usage but running it in the background and catching it's IO as it can run for very long times
on large and/or extremely misbalanced clusters.  It's not a daemon in the traditional sense.
  It really is a convience script so that those that aren't familiar with bash don't have
to remember how to catch stdout/stderr, or use disown or whatever.   I'd be very surprised
if there actually was an init script.

It's fun to note that the start-balancer script only appears to be documented in the Balancer
javadoc and the only place that Javadoc is really exposed is on Cloudera's website. ;)

bq. I think there should be some generic fix for when "--daemon" isn't specified, because
of user expectations.

As stated above, user expectation is consistency.  No consistency will mean we'll also need
to remove the --daemon status capability since it will be unreliable.

> Shell scripts call hadoop_verify_logdir even when command is not started as daemon
> ----------------------------------------------------------------------------------
>                 Key: HADOOP-13717
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13717
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Andrew Wang
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the "default"
case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which specifies the log location
isn't even used here, since the command is being started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into {{hadoop_start_daemon_wrapper}}
instead, which does use the outfile.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message