hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12857) Rework hadoop-tools
Date Thu, 10 Mar 2016 19:43:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189837#comment-15189837
] 

Allen Wittenauer commented on HADOOP-12857:
-------------------------------------------


Manual run without unit tests:

| Vote |      Subsystem |  Runtime   | Comment
============================================================================
|   0  |        reexec  |  0m 21s    | Docker mode activated. 
|   0  |     shelldocs  |  0m 4s     | Shelldocs was not available. 
|  +1  |       @author  |  0m 0s     | The patch does not contain any @author 
|      |                |            | tags.
|  +1  |    test4tests  |  0m 0s     | The patch appears to include 6 new or 
|      |                |            | modified test files.
|   0  |        mvndep  |  3m 48s    | Maven dependency ordering for branch 
|  +1  |    mvninstall  |  11m 42s   | trunk passed 
|  +1  |       compile  |  13m 16s   | trunk passed 
|  +1  |       mvnsite  |  12m 55s   | trunk passed 
|  +1  |    mvneclipse  |  2m 37s    | trunk passed 
|  +1  |       javadoc  |  10m 23s   | trunk passed 
|   0  |        mvndep  |  0m 20s    | Maven dependency ordering for patch 
|  +1  |    mvninstall  |  23m 27s   | the patch passed 
|  +1  |       compile  |  11m 19s   | the patch passed 
|  +1  |         javac  |  11m 19s   | the patch passed 
|  +1  |       mvnsite  |  13m 4s    | the patch passed 
|  +1  |    mvneclipse  |  0m 57s    | the patch passed 
|  +1  |    shellcheck  |  0m 7s     | The applied patch generated 0 new + 94 
|      |                |            | unchanged - 5 fixed = 94 total (was 99)
|  +1  |    whitespace  |  0m 0s     | Patch has no whitespace issues. 
|  +1  |           xml  |  0m 5s     | The patch has no ill-formed XML file. 
|  +1  |       javadoc  |  10m 53s   | the patch passed 
|  +1  |    asflicense  |  0m 24s    | Patch does not generate ASF License 
|      |                |            | warnings.
|      |                |  116m 18s  | 

Manually running unit tests:
{code}
 hadoop-common-project/hadoop-common$ mvn test -DskipTests -Pshelltest

[INFO] --- maven-antrun-plugin:1.7:run (common-test-bats-driver) @ hadoop-common ---
[INFO] Executing tasks

main:
     [exec] Running bats -t hadoop_add_classpath.bats
     [exec] 1..11
     [exec] ok 1 hadoop_add_classpath (simple not exist)
     [exec] ok 2 hadoop_add_classpath (simple wildcard not exist)
     [exec] ok 3 hadoop_add_classpath (simple exist)
     [exec] ok 4 hadoop_add_classpath (simple wildcard exist)
     [exec] ok 5 hadoop_add_classpath (simple dupecheck)
     [exec] ok 6 hadoop_add_classpath (default order)
     [exec] ok 7 hadoop_add_classpath (after order)
     [exec] ok 8 hadoop_add_classpath (before order)
     [exec] ok 9 hadoop_add_classpath (simple dupecheck 2)
     [exec] ok 10 hadoop_add_classpath (dupecheck 3)
     [exec] ok 11 hadoop_add_classpath (complex ordering)
     [exec] Running bats -t hadoop_add_colonpath.bats
     [exec] 1..9
     [exec] ok 1 hadoop_add_colonpath (simple not exist)
     [exec] ok 2 hadoop_add_colonpath (simple exist)
     [exec] ok 3 hadoop_add_colonpath (simple dupecheck)
     [exec] ok 4 hadoop_add_colonpath (default order)
     [exec] ok 5 hadoop_add_colonpath (after order)
     [exec] ok 6 hadoop_add_colonpath (before order)
     [exec] ok 7 hadoop_add_colonpath (simple dupecheck 2)
     [exec] ok 8 hadoop_add_colonpath (dupecheck 3)
     [exec] ok 9 hadoop_add_colonpath (complex ordering)
     [exec] Running bats -t hadoop_add_common_to_classpath.bats
     [exec] 1..3
     [exec] ok 1 hadoop_add_common_to_classpath (negative)
     [exec] ok 2 hadoop_add_common_to_classpath (positive)
     [exec] ok 3 hadoop_add_common_to_classpath (build paths)
     [exec] Running bats -t hadoop_add_javalibpath.bats
     [exec] 1..9
     [exec] ok 1 hadoop_add_javalibpath (simple not exist)
     [exec] ok 2 hadoop_add_javalibpath (simple exist)
     [exec] ok 3 hadoop_add_javalibpath (simple dupecheck)
     [exec] ok 4 hadoop_add_javalibpath (default order)
     [exec] ok 5 hadoop_add_javalibpath (after order)
     [exec] ok 6 hadoop_add_javalibpath (before order)
     [exec] ok 7 hadoop_add_javalibpath (simple dupecheck 2)
     [exec] ok 8 hadoop_add_javalibpath (dupecheck 3)
     [exec] ok 9 hadoop_add_javalibpath (complex ordering)
     [exec] Running bats -t hadoop_add_ldlibpath.bats
     [exec] 1..9
     [exec] ok 1 hadoop_add_ldlibpath (simple not exist)
     [exec] ok 2 hadoop_add_ldlibpath (simple exist)
     [exec] ok 3 hadoop_add_ldlibpath (simple dupecheck)
     [exec] ok 4 hadoop_add_ldlibpath (default order)
     [exec] ok 5 hadoop_add_ldlibpath (after order)
     [exec] ok 6 hadoop_add_ldlibpath (before order)
     [exec] ok 7 hadoop_add_ldlibpath (simple dupecheck 2)
     [exec] ok 8 hadoop_add_ldlibpath (dupecheck 3)
     [exec] ok 9 hadoop_add_ldlibpath (complex ordering)
     [exec] Running bats -t hadoop_add_param.bats
     [exec] 1..4
     [exec] ok 1 hadoop_add_param (positive 1)
     [exec] ok 2 hadoop_add_param (negative)
     [exec] ok 3 hadoop_add_param (positive 2)
     [exec] ok 4 hadoop_add_param (positive 3)
     [exec] Running bats -t hadoop_add_to_classpath_tools.bats
     [exec] 1..3
     [exec] ok 1 hadoop_classpath_tools (load)
     [exec] ok 2 hadoop_classpath_tools (not exist)
     [exec] ok 3 hadoop_classpath_tools (function)
     [exec] Running bats -t hadoop_add_to_classpath_userpath.bats
     [exec] 1..7
     [exec] ok 1 hadoop_add_to_classpath_userpath (nothing)
     [exec] ok 2 hadoop_add_to_classpath_userpath (none)
     [exec] ok 3 hadoop_add_to_classpath_userpath (only)
     [exec] ok 4 hadoop_add_to_classpath_userpath (classloader)
     [exec] ok 5 hadoop_add_to_classpath_userpath (1+1 dupe)
     [exec] ok 6 hadoop_add_to_classpath_userpath (3+2 after)
     [exec] ok 7 hadoop_add_to_classpath_userpath (3+2 before)
     [exec] Running bats -t hadoop_basic_init.bats
     [exec] 1..3
     [exec] ok 1 hadoop_basic_init (bad dir errors)
     [exec] ok 2 hadoop_basic_init (no non-dir overrides)
     [exec] ok 3 hadoop_basic_init (test non-dir overrides)
     [exec] Running bats -t hadoop_bootstrap.bats
     [exec] 1..2
     [exec] ok 1 hadoop_bootstrap (no libexec)
     [exec] ok 2 hadoop_bootstrap (libexec)
     [exec] Running bats -t hadoop_confdir.bats
     [exec] 1..9
     [exec] ok 1 hadoop_find_confdir (default)
     [exec] ok 2 hadoop_find_confdir (bw compat: conf)
     [exec] ok 3 hadoop_find_confdir (etc/hadoop)
     [exec] ok 4 hadoop_verify_confdir (negative) 
     [exec] ok 5 hadoop_verify_confdir (positive) 
     [exec] ok 6 hadoop_exec_hadoopenv (positive) 
     [exec] ok 7 hadoop_exec_hadoopenv (negative) 
     [exec] ok 8 hadoop_exec_userfuncs
     [exec] ok 9 hadoop_exec_hadooprc
     [exec] Running bats -t hadoop_deprecate_envvar.bats
     [exec] 1..2
     [exec] ok 1 hadoop_deprecate_envvar (replace)
     [exec] ok 2 hadoop_deprecate_envvar (no replace)
     [exec] Running bats -t hadoop_entry_tests.bats
     [exec] 1..4
     [exec] ok 1 hadoop_add_entry (positive 1)
     [exec] ok 2 hadoop_add_entry (negative)
     [exec] ok 3 hadoop_add_entry (positive 2)
     [exec] ok 4 hadoop_add_entry (positive 3)
     [exec] Running bats -t hadoop_finalize.bats
     [exec] 1..11
     [exec] ok 1 hadoop_finalize (shellprofiles)
     [exec] ok 2 hadoop_finalize (classpath)
     [exec] ok 3 hadoop_finalize (libpaths)
     [exec] ok 4 hadoop_finalize (heap)
     [exec] ok 5 hadoop_finalize (opts)
     [exec] ok 6 hadoop_finalize (cygwin prefix)
     [exec] ok 7 hadoop_finalize (cygwin conf dir)
     [exec] ok 8 hadoop_finalize (cygwin common)
     [exec] ok 9 hadoop_finalize (cygwin hdfs)
     [exec] ok 10 hadoop_finalize (cygwin yarn)
     [exec] ok 11 hadoop_finalize (cygwin mapred)
     [exec] Running bats -t hadoop_finalize_catalina_opts.bats
     [exec] 1..2
     [exec] ok 1 hadoop_finalize_catalina_opts (raw)
     [exec] ok 2 # skip (catalina commands not supported under cygwin yet) hadoop_finalize_catalina_opts
(cygwin)
     [exec] Running bats -t hadoop_finalize_classpath.bats
     [exec] 1..4
     [exec] ok 1 hadoop_finalize_classpath (only conf dir)
     [exec] ok 2 hadoop_finalize_classpath (before conf dir)
     [exec] ok 3 hadoop_finalize_classpath (adds user)
     [exec] ok 4 hadoop_finalize_classpath (calls cygwin)
     [exec] Running bats -t hadoop_finalize_hadoop_heap.bats
     [exec] 1..8
     [exec] ok 1 hadoop_finalize_hadoop_heap (negative)
     [exec] ok 2 hadoop_finalize_hadoop_heap (no unit max)
     [exec] ok 3 hadoop_finalize_hadoop_heap (no unit old)
     [exec] ok 4 hadoop_finalize_hadoop_heap (unit max)
     [exec] ok 5 hadoop_finalize_hadoop_heap (unit old)
     [exec] ok 6 hadoop_finalize_hadoop_heap (no unit min)
     [exec] ok 7 hadoop_finalize_hadoop_heap (unit min)
     [exec] ok 8 hadoop_finalize_hadoop_heap (dedupe)
     [exec] Running bats -t hadoop_finalize_hadoop_opts.bats
     [exec] 1..2
     [exec] ok 1 hadoop_finalize_hadoop_opts (raw)
     [exec] ok 2 hadoop_finalize_hadoop_opts (cygwin)
     [exec] Running bats -t hadoop_finalize_libpaths.bats
     [exec] 1..2
     [exec] ok 1 hadoop_finalize_libpaths (negative)
     [exec] ok 2 hadoop_finalize_libpaths (positive)
     [exec] Running bats -t hadoop_java_setup.bats
     [exec] 1..4
     [exec] ok 1 hadoop_java_setup (negative not set)
     [exec] ok 2 hadoop_java_setup (negative not a dir)
     [exec] ok 3 hadoop_java_setup (negative not exec)
     [exec] ok 4 hadoop_java_setup (positive)
     [exec] Running bats -t hadoop_os_tricks.bats
     [exec] 1..3
     [exec] ok 1 hadoop_os_tricks (cygwin sets cygwin)
     [exec] ok 2 hadoop_os_tricks (linux sets arena max)
     [exec] ok 3 hadoop_os_tricks (osx sets java_home)
     [exec] Running bats -t hadoop_rotate_log.bats
     [exec] 1..4
     [exec] ok 1 hadoop_rotate_log (defaults)
     [exec] ok 2 hadoop_rotate_log (one archive log)
     [exec] ok 3 hadoop_rotate_log (default five archive logs)
     [exec] ok 4 hadoop_rotate_log (ten archive logs)
     [exec] Running bats -t hadoop_shellprofile.bats
     [exec] 1..9
     [exec] ok 1 hadoop_import_shellprofiles (negative)
     [exec] ok 2 hadoop_import_shellprofiles (libexec sh import)
     [exec] ok 3 hadoop_import_shellprofiles (libexec conf sh import+override)
     [exec] ok 4 hadoop_import_shellprofiles (libexec no cmd import)
     [exec] ok 5 hadoop_import_shellprofiles (H_O_T)
     [exec] ok 6 hadoop_add_profile+hadoop_shellprofiles_init
     [exec] ok 7 hadoop_add_profile+hadoop_shellprofiles_classpath
     [exec] ok 8 hadoop_add_profile+hadoop_shellprofiles_nativelib
     [exec] ok 9 hadoop_add_profile+hadoop_shellprofiles_finalize
     [exec] Running bats -t hadoop_slaves.bats
     [exec] 1..3
     [exec] ok 1 hadoop_populate_slaves_file (specific file)
     [exec] ok 2 hadoop_populate_slaves_file (specific conf dir file)
     [exec] ok 3 hadoop_populate_slaves_file (no file)
     [exec] Running bats -t hadoop_ssh.bats
     [exec] 1..7
     [exec] ok 1 # skip (Not implemented) hadoop_actual_ssh
     [exec] ok 2 # skip (Not implemented) hadoop_connect_to_hosts
     [exec] ok 3 # skip (Not implemented) hadoop_connect_to_hosts_without_pdsh
     [exec] ok 4 hadoop_common_slave_mode_execute (--slaves 1)
     [exec] ok 5 hadoop_common_slave_mode_execute (--slaves 2)
     [exec] ok 6 hadoop_common_slave_mode_execute (--hosts)
     [exec] ok 7 hadoop_common_slave_mode_execute (--hostnames 2)
     [exec] Running bats -t hadoop_stop_daemon.bats
     [exec] 1..1
     [exec] ok 1 hadoop_stop_daemon
     [exec] Running bats -t hadoop_stop_secure_daemon.bats
     [exec] 1..1
     [exec] ok 1 hadoop_stop_secure_daemon
     [exec] Running bats -t hadoop_translate_cygwin_path.bats
     [exec] 1..3
     [exec] ok 1 hadoop_translate_cygwin_path (negative)
     [exec] ok 2 hadoop_translate_cygwin_path (positive)
     [exec] ok 3 hadoop_translate_cygwin_path (path positive)
     [exec] Running bats -t hadoop_validate_classname.bats
     [exec] 1..2
     [exec] ok 1 hadoop_validate_classname (negative)
     [exec] ok 2 hadoop_validate_classname (positive)
[INFO] Executed tasks
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51.188s
[INFO] Finished at: Thu Mar 10 11:42:32 PST 2016
[INFO] Final Memory: 26M/84M
[INFO] ------------------------------------------------------------------------
{code}


> Rework hadoop-tools
> -------------------
>
>                 Key: HADOOP-12857
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12857
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>         Attachments: HADOOP-12857.00.patch, HADOOP-12857.01.patch, HADOOP-12857.02.patch
>
>
> As hadoop-tools grows bigger and bigger, it's becoming evident that having a single directory
that gets sucked in is starting to become a big burden as the number of tools grows.  Let's
rework this to be smarter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message