hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9902) Shell script rewrite
Date Mon, 16 Sep 2013 20:26:56 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768713#comment-13768713
] 

Allen Wittenauer commented on HADOOP-9902:
------------------------------------------

Since I'm getting ready to post a patch, how about an 'end result' example!  Here is the comamnd
line for the resource manager from my real, 100+ node test grid.

Before the changes:
{code}
/usr/java/default/bin/java
-Dproc_resourcemanager
-Xmx1000m
-Xmx24g
-Dyarn.server.resourcemanager.appsummary.log.file=rm-appsummary.log
-Dyarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY
-Xloggc:/export/apps/hadoop/logs/gc-nn.log-201308261726
-Dcom.sun.management.jmxremote.port=9010
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dhadoop.log.dir=/export/apps/hadoop/logs
-Dyarn.log.dir=/export/apps/hadoop/logs
-Dhadoop.log.file=yarn-yarn-resourcemanager-eat1-hcl4083.grid.linkedin.com.log
-Dyarn.log.file=yarn-yarn-resourcemanager-eat1-hcl4083.grid.linkedin.com.log
-Dyarn.home.dir=
-Dyarn.id.str=yarn
-Dhadoop.root.logger=INFO,DRFA
-Dyarn.root.logger=INFO,DRFA
-Djava.library.path=/export/apps/hadoop/latest/lib/native
-Dyarn.policy.file=hadoop-policy.xml
-Dhadoop.log.dir=/export/apps/hadoop/logs
-Dyarn.log.dir=/export/apps/hadoop/logs
-Dhadoop.log.file=yarn-yarn-resourcemanager-eat1-hcl4083.grid.linkedin.com.log
-Dyarn.log.file=yarn-yarn-resourcemanager-eat1-hcl4083.grid.linkedin.com.log
-Dyarn.home.dir=/export/apps/hadoop/latest
-Dhadoop.home.dir=/export/apps/hadoop/latest
-Dhadoop.root.logger=INFO,DRFA
-Dyarn.root.logger=INFO,DRFA
-Djava.library.path=/export/apps/hadoop/latest/lib/native
-classpath
/export/apps/hadoop/site/etc/hadoop
/export/apps/hadoop/site/etc/hadoop
/export/apps/hadoop/site/etc/hadoop
/export/apps/hadoop/latest/share/hadoop/common/lib/*
/export/apps/hadoop/latest/share/hadoop/common/*
/export/apps/hadoop/latest/share/hadoop/hdfs
/export/apps/hadoop/latest/share/hadoop/hdfs/lib/*
/export/apps/hadoop/latest/share/hadoop/hdfs/*
/export/apps/hadoop/latest/share/hadoop/yarn/lib/*
/export/apps/hadoop/latest/share/hadoop/yarn/*
/export/apps/hadoop/latest/share/hadoop/mapreduce/lib/*
/export/apps/hadoop/latest/share/hadoop/mapreduce/*
/export/apps/hadoop/site/lib/grid-topology-1.0.jar
/export/apps/hadoop/latest/contrib/capacity-scheduler/*.jar
/export/apps/hadoop/site/lib/grid-topology-1.0.jar
/export/apps/hadoop/latest/contrib/capacity-scheduler/*.jar
/export/apps/hadoop/site/lib/grid-topology-1.0.jar
/export/apps/hadoop/latest/contrib/capacity-scheduler/*.jar
/export/apps/hadoop/latest/share/hadoop/yarn/*
/export/apps/hadoop/latest/share/hadoop/yarn/lib/*
/export/apps/hadoop/site/etc/hadoop/rm-config/log4j.properties
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
{code}

After the changes:
{code}
/usr/java/default/bin/java
-Dproc_resourcemanager
-Xloggc:/export/apps/hadoop/logs/gc-nn.log-201309162014
-Dcom.sun.management.jmxremote.port=9010
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Xmx24g
-Dyarn.server.resourcemanager.appsummary.log.file=rm-appsummary.log
-Dyarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY
-Dyarn.log.dir=/export/apps/hadoop/logs
-Dyarn.log.file=yarn-yarn-resourcemanager-eat1-hcl4083.grid.linkedin.com.log
-Dyarn.home.dir=/export/apps/hadoop/latest
-Dyarn.root.logger=INFO,DRFA
-Djava.library.path=/export/apps/hadoop/latest/lib/native
-Dhadoop.log.dir=/export/apps/hadoop/logs
-Dhadoop.log.file=yarn-yarn-resourcemanager-eat1-hcl4083.grid.linkedin.com.log
-Dhadoop.home.dir=/export/apps/hadoop/latest
-Dhadoop.id.str=yarn
-Dhadoop.root.logger=INFO,DRFA
-Dhadoop.policy.file=hadoop-policy.xml
-Dhadoop.security.logger=INFO,NullAppender
-Djava.net.preferIPv4Stack=true
-classpath
/export/apps/hadoop/site/lib/grid-topology-1.0.jar
/export/apps/hadoop/latest/contrib/capacity-scheduler/*.jar
/export/apps/hadoop/site/etc/hadoop
/export/apps/hadoop/latest/share/hadoop/common/lib/*
/export/apps/hadoop/latest/share/hadoop/common/*
/export/apps/hadoop/latest/share/hadoop/hdfs
/export/apps/hadoop/latest/share/hadoop/hdfs/lib/*
/export/apps/hadoop/latest/share/hadoop/hdfs/*
/export/apps/hadoop/latest/share/hadoop/yarn/lib/*
/export/apps/hadoop/latest/share/hadoop/yarn/*
/export/apps/hadoop/latest/share/hadoop/mapreduce/lib/*
/export/apps/hadoop/latest/share/hadoop/mapreduce/*
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
{code}

2500 bytes vs. 1750 bytes, almost all the savings are from the classpath.

There are still a few problems with the 'after' output but... they are mainly from my local
config and not coming from the scripts. :)
                
> Shell script rewrite
> --------------------
>
>                 Key: HADOOP-9902
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9902
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 2.1.1-beta
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>         Attachments: more-info.txt, scripts.tgz
>
>
> Umbrella JIRA for shell script rewrite.  See more-info.txt for more details.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message