hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran" <pjayachand...@hortonworks.com>
Subject Re: Getting Tez working against cdh 5.3
Date Tue, 20 Jan 2015 21:47:36 GMT
Hi Ed,




Can you post your concerns to user@tez.apache.org? Someone should be able to help you out
with the issue.





- Prasanth

On Tue, Jan 20, 2015 at 12:35 PM, Edward Capriolo <edlinuxguru@gmail.com>
wrote:

> Actually more likely something like this:
> https://issues.apache.org/jira/browse/TEZ-1621
> On Tue, Jan 20, 2015 at 2:02 PM, Prasanth Jayachandran <
> pjayachandran@hortonworks.com> wrote:
>> My guess is..
>>  "java" binary is not in PATH of the shell script that launches the
>> container.. try creating a symbolic link in /bin/ to point to java..
>>
>> On Tue, Jan 20, 2015 at 7:22 AM, Edward Capriolo <edlinuxguru@gmail.com>
>> wrote:
>>
>>> It seems that CDH does not ship with enough jars to run tez out of the
>>> box.
>>>
>>> I have found the related cloudera forked pom.
>>>
>>> In this pom hive is built against tez 0.4.1-incubating-tez2.0-SHAPSHOT
>>>
>>> Thus I followed the instructions here:
>>>
>>> http://tez.apache.org/install_pre_0_5_0.html
>>>
>>> hive> dfs -lsr /apps
>>>     > ;
>>> lsr: DEPRECATED: Please use 'ls -R' instead.
>>> drwxr-xr-x   - ecapriolo supergroup          0 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating
>>> drwxr-xr-x   - ecapriolo supergroup          0 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib
>>> -rw-r--r--   3 ecapriolo supergroup     303139 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/avro-1.7.4.jar
>>> -rw-r--r--   3 ecapriolo supergroup      41123 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/commons-cli-1.2.jar
>>> -rw-r--r--   3 ecapriolo supergroup     610259 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/commons-collections4-4.0.jar
>>> -rw-r--r--   3 ecapriolo supergroup    1648200 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/guava-11.0.2.jar
>>> -rw-r--r--   3 ecapriolo supergroup     710492 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/guice-3.0.jar
>>> -rw-r--r--   3 ecapriolo supergroup     656365 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-common-2.2.0.jar
>>> -rw-r--r--   3 ecapriolo supergroup    1455001 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-core-2.2.0.jar
>>> -rw-r--r--   3 ecapriolo supergroup      21537 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-shuffle-2.2.0.jar
>>> -rw-r--r--   3 ecapriolo supergroup      81743 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/jettison-1.3.4.jar
>>> -rw-r--r--   3 ecapriolo supergroup     533455 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/protobuf-java-2.5.0.jar
>>> -rw-r--r--   3 ecapriolo supergroup     995968 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/lib/snappy-java-1.0.4.1.jar
>>> -rw-r--r--   3 ecapriolo supergroup     752332 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/tez-api-0.4.1-incubating.jar
>>> -rw-r--r--   3 ecapriolo supergroup      34089 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/tez-common-0.4.1-incubating.jar
>>> -rw-r--r--   3 ecapriolo supergroup     980132 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/tez-dag-0.4.1-incubating.jar
>>> -rw-r--r--   3 ecapriolo supergroup     246395 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/tez-mapreduce-0.4.1-incubating.jar
>>> -rw-r--r--   3 ecapriolo supergroup     199984 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/tez-mapreduce-examples-0.4.1-incubating.jar
>>> -rw-r--r--   3 ecapriolo supergroup     114676 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/tez-runtime-internals-0.4.1-incubating.jar
>>> -rw-r--r--   3 ecapriolo supergroup     352835 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/tez-runtime-library-0.4.1-incubating.jar
>>> -rw-r--r--   3 ecapriolo supergroup       6832 2015-01-16 23:00
>>> /apps/tez-0.4.1-incubating/tez-tests-0.4.1-incubating.jar
>>>
>>> This is my tez-site.xml
>>>
>>> <configuration>
>>>   <property>
>>>     <name>tez.lib.uris</name>
>>>     <value>${fs.default.name}/apps/tez-0.4.1-incubating,${fs.default.name
>>> }/apps/tez-0.4.1-incubating/lib/</value>
>>>   </property>
>>> </configuration>
>>>
>>> [ecapriolo@production-hadoop-cdh-69-7 ~]$ ls -lahR
>>> /home/ecapriolo/tez-0.4.1-incubating/
>>> /home/ecapriolo/tez-0.4.1-incubating/:
>>> total 2.7M
>>> drwxrwxr-x 3 ecapriolo ecapriolo 4.0K Jan 16 22:54 .
>>> drwx------ 7 ecapriolo ecapriolo  20K Jan 20 15:20 ..
>>> drwxrwxr-x 2 ecapriolo ecapriolo 4.0K Jan 16 22:54 lib
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 735K Jan 16 22:54
>>> tez-api-0.4.1-incubating.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo  34K Jan 16 22:54
>>> tez-common-0.4.1-incubating.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 958K Jan 16 22:54
>>> tez-dag-0.4.1-incubating.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 241K Jan 16 22:54
>>> tez-mapreduce-0.4.1-incubating.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 196K Jan 16 22:54
>>> tez-mapreduce-examples-0.4.1-incubating.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 112K Jan 16 22:54
>>> tez-runtime-internals-0.4.1-incubating.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 345K Jan 16 22:54
>>> tez-runtime-library-0.4.1-incubating.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 6.7K Jan 16 22:54
>>> tez-tests-0.4.1-incubating.jar
>>>
>>> /home/ecapriolo/tez-0.4.1-incubating/lib:
>>> total 6.8M
>>> drwxrwxr-x 2 ecapriolo ecapriolo 4.0K Jan 16 22:54 .
>>> drwxrwxr-x 3 ecapriolo ecapriolo 4.0K Jan 16 22:54 ..
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 297K Jan 16 22:54 avro-1.7.4.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo  41K Jan 16 22:54 commons-cli-1.2.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 596K Jan 16 22:54
>>> commons-collections4-4.0.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 1.6M Jan 16 22:54 guava-11.0.2.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 694K Jan 16 22:54 guice-3.0.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 641K Jan 16 22:54
>>> hadoop-mapreduce-client-common-2.2.0.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 1.4M Jan 16 22:54
>>> hadoop-mapreduce-client-core-2.2.0.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo  22K Jan 16 22:54
>>> hadoop-mapreduce-client-shuffle-2.2.0.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo  80K Jan 16 22:54 jettison-1.3.4.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 521K Jan 16 22:54 protobuf-java-2.5.0.jar
>>> -rw-rw-r-- 1 ecapriolo ecapriolo 973K Jan 16 22:54 snappy-java-1.0.4.1.jar
>>>
>>> tez.sh
>>> TEZ_CONF_DIR=/home/ecapriolo
>>> TEZ_JARS=/home/ecapriolo/tez-0.4.1-incubating
>>> export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
>>> #hive -hiveconf mapreduce.framework.name=yarn-tez
>>> #hive -hiveconf hive.root.logger=DEGUG,console
>>> hive
>>>
>>> hive> set
>>> hive.execution.engine=tez;
>>> hive> select sum(viral_count) from author_article_hourly where dt=
>>> 2015011622;
>>> Total jobs = 1
>>> Launching Job 1 out of 1
>>>
>>>
>>> Status: Running (application id: application_1420748315294_70716)
>>>
>>> Map 1: -/-    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Map 1: 0/1    Reducer 2: 0/1
>>> Status: Failed
>>> Vertex failed, vertexName=Map 1,
>>> vertexId=vertex_1420748315294_70716_1_01, diagnostics=[Task failed,
>>> taskId=task_1420748315294_70716_1_01_000000,
>>> diagnostics=[AttemptID:attempt_1420748315294_70716_1_01_000000_0
>>> Info:Container container_1420748315294_70716_01_000002 COMPLETED with
>>> diagnostics set to [Exception from container-launch.
>>> Container id: container_1420748315294_70716_01_000002
>>> Exit code: 255
>>> Stack trace: ExitCodeException exitCode=255:
>>>     at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>     at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>     at
>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>     at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>>>     at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>>>     at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>     at java.lang.Thread.run(Thread.java:745)
>>>
>>> Other logs are here (from the containers that I can extract from yawn.
>>>
>>> Exit code: 255
>>> Stack trace: ExitCodeException exitCode=255:
>>>     at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>     at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>     at
>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>     at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>>>     at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>>>     at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>     at java.lang.Thread.run(Thread.java:744)
>>>
>>> application_1420748315294_70716
>>>
>>> digraph _1 {
>>> graph [ label="_1", fontsize=24, fontname=Helvetica];
>>> node [fontsize=12, fontname=Helvetica];
>>> edge [fontsize=9, fontcolor=blue, fontname=Arial];
>>> "_1.Map_1_author_article_hourly" [ label =
>>> "Map_1[author_article_hourly]", shape = "box" ];
>>> "_1.Map_1_author_article_hourly" -> "_1.Map_1" [ label = "Input
>>> [inputClass=MRInputLegacy,\n initializer=MRInputAMSplitGenerator]" ];
>>> "_1.Reducer_2" [ label = "Reducer_2[ReduceTezProcessor]" ];
>>> "_1.Reducer_2" -> "_1.Reducer_2_out_Reducer_2" [ label = "Output
>>> [outputClass=MROutput,\n initializer=]" ];
>>> "_1.Map_1" [ label = "Map_1[MapTezProcessor]" ];
>>> "_1.Map_1" -> "_1.Reducer_2" [ label = "[input=OnFileSortedOutput,\n
>>> output=ShuffledMergedInputLegacy,\n dataMovement=SCATTER_GATHER,\n
>>> schedulingType=SEQUENTIAL]" ];
>>> "_1.Reducer_2_out_Reducer_2" [ label = "Reducer_2[out_Reducer_2]", shape
>>> = "box" ];
>>> }
>>>
>>>
>>> Container exited with a non-zero exit code 255
>>> ]], Vertex failed as one or more tasks failed. failedTasks:1]
>>> Vertex killed, vertexName=Reducer 2,
>>> vertexId=vertex_1420748315294_70716_1_00, diagnostics=[Vertex received Kill
>>> while in RUNNING state., Vertex killed as other vertex failed.
>>> failedTasks:0]
>>> DAG failed due to vertex failure. failedVertices:1 killedVertices:1,
>>> counters=Counters: 2, org.apache.tez.common.counters.DAGCounter,
>>> NUM_FAILED_TASKS=4, TOTAL_LAUNCHED_TASKS=4
>>> 2015-01-20 15:02:21,934 INFO [AsyncDispatcher event handler]
>>> org.apache.tez.dag.app.dag.impl.DAGImpl: DAG: dag_1420748315294_70716_1
>>> finished with state: FAILED
>>> 2015-01-20 15:02:21,934 INFO [AsyncDispatcher event handler]
>>> org.apache.tez.dag.app.dag.impl.DAGImpl: dag_1420748315294_70716_1
>>> transitioned from TERMINATING to FAILED
>>> 2015-01-20 15:02:21,935 INFO [AsyncDispatcher event handler]
>>> org.apache.tez.dag.app.DAGAppMaster: DAG completed,
>>> dagId=dag_1420748315294_70716_1, dagState=FAILED
>>> 2015-01-20 15:02:21,935 INFO [AsyncDispatcher event handler]
>>> org.apache.tez.common.TezUtils: Redirecting log files based on addend:
>>> dag_1420748315294_70716_1_post
>>>
>>> Has anyone got this working or have and ideas as to what is up here?
>>>
>>> Thanks,
>>> Edward
>>>
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message