hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13717) Shell scripts call hadoop_verify_logdir even when command is not started as daemon
Date Thu, 13 Oct 2016 17:23:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572596#comment-15572596

Allen Wittenauer commented on HADOOP-13717:

Ah, OK. 

bq. Is it reasonable to also skip these checks in this situation?

In the general daemon case we definitely need to leave the hadoop_verify_logdir in place.
 This will get followed up with verifying the pid dir as well.  This is needed because many
folks run the daemon commands "interactively" from a shell code perspective but do their own
daemonization from init.  There's also the docker case where individual daemons are in their
own containers.   etc, etc. We're almost always better checking for the existence of this
dirs rather than letting Java go haywire.

But balancer (and mover and probably a handful of others) are, in many ways, special.

We effectively have a problem of history. sbin/start-balancer.sh was added way back when as
a convenience  to background the balancer.  When the rewrite happened, balancer was added
to the commands that could be daemonized as a result.   

There might also be a privilege problem here too.  I don't think balancer can be run as by
a non-privileged user, usually making the log and pid dir verification (usually) safe and
somewhat required if we want all of our daemons to function in the same way (consistency!).
 The pid check at the UNIX level definitely helps prevent multiple instances running on the
same host way faster (and somewhat more reliably!) than the Java checks. 

To me, the questions become:

a) should balancer stay as a daemon? what about the other not-really-daemons-but-can-run-in-daemon-mode
b) should instead there be logic to say that failing logdir and piddir are optionally fatal?
c) should a local override be used instead? (e.g., define a shell profile that replaces the
hdfs balancer command)

> Shell scripts call hadoop_verify_logdir even when command is not started as daemon
> ----------------------------------------------------------------------------------
>                 Key: HADOOP-13717
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13717
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Andrew Wang
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the "default"
case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which specifies the log location
isn't even used here, since the command is being started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into {{hadoop_start_daemon_wrapper}}
instead, which does use the outfile.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message