hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Getting Tez working against cdh 5.3
Date Tue, 20 Jan 2015 20:34:20 GMT
Actually more likely something like this:

https://issues.apache.org/jira/browse/TEZ-1621

On Tue, Jan 20, 2015 at 2:02 PM, Prasanth Jayachandran <
pjayachandran@hortonworks.com> wrote:

> My guess is..
>  "java" binary is not in PATH of the shell script that launches the
> container.. try creating a symbolic link in /bin/ to point to java..
>
> On Tue, Jan 20, 2015 at 7:22 AM, Edward Capriolo <edlinuxguru@gmail.com>
> wrote:
>
>> It seems that CDH does not ship with enough jars to run tez out of the
>> box.
>>
>> I have found the related cloudera forked pom.
>>
>> In this pom hive is built against tez 0.4.1-incubating-tez2.0-SHAPSHOT
>>
>> Thus I followed the instructions here:
>>
>> http://tez.apache.org/install_pre_0_5_0.html
>>
>> hive> dfs -lsr /apps
>>     > ;
>> lsr: DEPRECATED: Please use 'ls -R' instead.
>> drwxr-xr-x   - ecapriolo supergroup          0 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating
>> drwxr-xr-x   - ecapriolo supergroup          0 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib
>> -rw-r--r--   3 ecapriolo supergroup     303139 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/avro-1.7.4.jar
>> -rw-r--r--   3 ecapriolo supergroup      41123 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/commons-cli-1.2.jar
>> -rw-r--r--   3 ecapriolo supergroup     610259 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/commons-collections4-4.0.jar
>> -rw-r--r--   3 ecapriolo supergroup    1648200 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/guava-11.0.2.jar
>> -rw-r--r--   3 ecapriolo supergroup     710492 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/guice-3.0.jar
>> -rw-r--r--   3 ecapriolo supergroup     656365 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-common-2.2.0.jar
>> -rw-r--r--   3 ecapriolo supergroup    1455001 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-core-2.2.0.jar
>> -rw-r--r--   3 ecapriolo supergroup      21537 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-shuffle-2.2.0.jar
>> -rw-r--r--   3 ecapriolo supergroup      81743 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/jettison-1.3.4.jar
>> -rw-r--r--   3 ecapriolo supergroup     533455 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/protobuf-java-2.5.0.jar
>> -rw-r--r--   3 ecapriolo supergroup     995968 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/lib/snappy-java-1.0.4.1.jar
>> -rw-r--r--   3 ecapriolo supergroup     752332 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/tez-api-0.4.1-incubating.jar
>> -rw-r--r--   3 ecapriolo supergroup      34089 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/tez-common-0.4.1-incubating.jar
>> -rw-r--r--   3 ecapriolo supergroup     980132 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/tez-dag-0.4.1-incubating.jar
>> -rw-r--r--   3 ecapriolo supergroup     246395 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/tez-mapreduce-0.4.1-incubating.jar
>> -rw-r--r--   3 ecapriolo supergroup     199984 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/tez-mapreduce-examples-0.4.1-incubating.jar
>> -rw-r--r--   3 ecapriolo supergroup     114676 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/tez-runtime-internals-0.4.1-incubating.jar
>> -rw-r--r--   3 ecapriolo supergroup     352835 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/tez-runtime-library-0.4.1-incubating.jar
>> -rw-r--r--   3 ecapriolo supergroup       6832 2015-01-16 23:00
>> /apps/tez-0.4.1-incubating/tez-tests-0.4.1-incubating.jar
>>
>> This is my tez-site.xml
>>
>> <configuration>
>>   <property>
>>     <name>tez.lib.uris</name>
>>     <value>${fs.default.name}/apps/tez-0.4.1-incubating,${fs.default.name
>> }/apps/tez-0.4.1-incubating/lib/</value>
>>   </property>
>> </configuration>
>>
>> [ecapriolo@production-hadoop-cdh-69-7 ~]$ ls -lahR
>> /home/ecapriolo/tez-0.4.1-incubating/
>> /home/ecapriolo/tez-0.4.1-incubating/:
>> total 2.7M
>> drwxrwxr-x 3 ecapriolo ecapriolo 4.0K Jan 16 22:54 .
>> drwx------ 7 ecapriolo ecapriolo  20K Jan 20 15:20 ..
>> drwxrwxr-x 2 ecapriolo ecapriolo 4.0K Jan 16 22:54 lib
>> -rw-rw-r-- 1 ecapriolo ecapriolo 735K Jan 16 22:54
>> tez-api-0.4.1-incubating.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo  34K Jan 16 22:54
>> tez-common-0.4.1-incubating.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 958K Jan 16 22:54
>> tez-dag-0.4.1-incubating.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 241K Jan 16 22:54
>> tez-mapreduce-0.4.1-incubating.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 196K Jan 16 22:54
>> tez-mapreduce-examples-0.4.1-incubating.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 112K Jan 16 22:54
>> tez-runtime-internals-0.4.1-incubating.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 345K Jan 16 22:54
>> tez-runtime-library-0.4.1-incubating.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 6.7K Jan 16 22:54
>> tez-tests-0.4.1-incubating.jar
>>
>> /home/ecapriolo/tez-0.4.1-incubating/lib:
>> total 6.8M
>> drwxrwxr-x 2 ecapriolo ecapriolo 4.0K Jan 16 22:54 .
>> drwxrwxr-x 3 ecapriolo ecapriolo 4.0K Jan 16 22:54 ..
>> -rw-rw-r-- 1 ecapriolo ecapriolo 297K Jan 16 22:54 avro-1.7.4.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo  41K Jan 16 22:54 commons-cli-1.2.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 596K Jan 16 22:54
>> commons-collections4-4.0.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 1.6M Jan 16 22:54 guava-11.0.2.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 694K Jan 16 22:54 guice-3.0.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 641K Jan 16 22:54
>> hadoop-mapreduce-client-common-2.2.0.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 1.4M Jan 16 22:54
>> hadoop-mapreduce-client-core-2.2.0.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo  22K Jan 16 22:54
>> hadoop-mapreduce-client-shuffle-2.2.0.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo  80K Jan 16 22:54 jettison-1.3.4.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 521K Jan 16 22:54 protobuf-java-2.5.0.jar
>> -rw-rw-r-- 1 ecapriolo ecapriolo 973K Jan 16 22:54 snappy-java-1.0.4.1.jar
>>
>> tez.sh
>> TEZ_CONF_DIR=/home/ecapriolo
>> TEZ_JARS=/home/ecapriolo/tez-0.4.1-incubating
>> export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
>> #hive -hiveconf mapreduce.framework.name=yarn-tez
>> #hive -hiveconf hive.root.logger=DEGUG,console
>> hive
>>
>> hive> set
>> hive.execution.engine=tez;
>> hive> select sum(viral_count) from author_article_hourly where dt=
>> 2015011622;
>> Total jobs = 1
>> Launching Job 1 out of 1
>>
>>
>> Status: Running (application id: application_1420748315294_70716)
>>
>> Map 1: -/-    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Map 1: 0/1    Reducer 2: 0/1
>> Status: Failed
>> Vertex failed, vertexName=Map 1,
>> vertexId=vertex_1420748315294_70716_1_01, diagnostics=[Task failed,
>> taskId=task_1420748315294_70716_1_01_000000,
>> diagnostics=[AttemptID:attempt_1420748315294_70716_1_01_000000_0
>> Info:Container container_1420748315294_70716_01_000002 COMPLETED with
>> diagnostics set to [Exception from container-launch.
>> Container id: container_1420748315294_70716_01_000002
>> Exit code: 255
>> Stack trace: ExitCodeException exitCode=255:
>>     at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>     at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>     at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>     at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>>     at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>>     at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>     at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>     at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>     at java.lang.Thread.run(Thread.java:745)
>>
>> Other logs are here (from the containers that I can extract from yawn.
>>
>> Exit code: 255
>> Stack trace: ExitCodeException exitCode=255:
>>     at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>     at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>     at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>     at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>>     at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>>     at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>     at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>     at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>     at java.lang.Thread.run(Thread.java:744)
>>
>> application_1420748315294_70716
>>
>> digraph _1 {
>> graph [ label="_1", fontsize=24, fontname=Helvetica];
>> node [fontsize=12, fontname=Helvetica];
>> edge [fontsize=9, fontcolor=blue, fontname=Arial];
>> "_1.Map_1_author_article_hourly" [ label =
>> "Map_1[author_article_hourly]", shape = "box" ];
>> "_1.Map_1_author_article_hourly" -> "_1.Map_1" [ label = "Input
>> [inputClass=MRInputLegacy,\n initializer=MRInputAMSplitGenerator]" ];
>> "_1.Reducer_2" [ label = "Reducer_2[ReduceTezProcessor]" ];
>> "_1.Reducer_2" -> "_1.Reducer_2_out_Reducer_2" [ label = "Output
>> [outputClass=MROutput,\n initializer=]" ];
>> "_1.Map_1" [ label = "Map_1[MapTezProcessor]" ];
>> "_1.Map_1" -> "_1.Reducer_2" [ label = "[input=OnFileSortedOutput,\n
>> output=ShuffledMergedInputLegacy,\n dataMovement=SCATTER_GATHER,\n
>> schedulingType=SEQUENTIAL]" ];
>> "_1.Reducer_2_out_Reducer_2" [ label = "Reducer_2[out_Reducer_2]", shape
>> = "box" ];
>> }
>>
>>
>> Container exited with a non-zero exit code 255
>> ]], Vertex failed as one or more tasks failed. failedTasks:1]
>> Vertex killed, vertexName=Reducer 2,
>> vertexId=vertex_1420748315294_70716_1_00, diagnostics=[Vertex received Kill
>> while in RUNNING state., Vertex killed as other vertex failed.
>> failedTasks:0]
>> DAG failed due to vertex failure. failedVertices:1 killedVertices:1,
>> counters=Counters: 2, org.apache.tez.common.counters.DAGCounter,
>> NUM_FAILED_TASKS=4, TOTAL_LAUNCHED_TASKS=4
>> 2015-01-20 15:02:21,934 INFO [AsyncDispatcher event handler]
>> org.apache.tez.dag.app.dag.impl.DAGImpl: DAG: dag_1420748315294_70716_1
>> finished with state: FAILED
>> 2015-01-20 15:02:21,934 INFO [AsyncDispatcher event handler]
>> org.apache.tez.dag.app.dag.impl.DAGImpl: dag_1420748315294_70716_1
>> transitioned from TERMINATING to FAILED
>> 2015-01-20 15:02:21,935 INFO [AsyncDispatcher event handler]
>> org.apache.tez.dag.app.DAGAppMaster: DAG completed,
>> dagId=dag_1420748315294_70716_1, dagState=FAILED
>> 2015-01-20 15:02:21,935 INFO [AsyncDispatcher event handler]
>> org.apache.tez.common.TezUtils: Redirecting log files based on addend:
>> dag_1420748315294_70716_1_post
>>
>> Has anyone got this working or have and ideas as to what is up here?
>>
>> Thanks,
>> Edward
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message