hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Getting Tez working against cdh 5.3
Date Tue, 20 Jan 2015 15:22:30 GMT
It seems that CDH does not ship with enough jars to run tez out of the box.

I have found the related cloudera forked pom.

In this pom hive is built against tez 0.4.1-incubating-tez2.0-SHAPSHOT

Thus I followed the instructions here:

http://tez.apache.org/install_pre_0_5_0.html

hive> dfs -lsr /apps
    > ;
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x   - ecapriolo supergroup          0 2015-01-16 23:00
/apps/tez-0.4.1-incubating
drwxr-xr-x   - ecapriolo supergroup          0 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib
-rw-r--r--   3 ecapriolo supergroup     303139 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/avro-1.7.4.jar
-rw-r--r--   3 ecapriolo supergroup      41123 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/commons-cli-1.2.jar
-rw-r--r--   3 ecapriolo supergroup     610259 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/commons-collections4-4.0.jar
-rw-r--r--   3 ecapriolo supergroup    1648200 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/guava-11.0.2.jar
-rw-r--r--   3 ecapriolo supergroup     710492 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/guice-3.0.jar
-rw-r--r--   3 ecapriolo supergroup     656365 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-common-2.2.0.jar
-rw-r--r--   3 ecapriolo supergroup    1455001 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-core-2.2.0.jar
-rw-r--r--   3 ecapriolo supergroup      21537 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-shuffle-2.2.0.jar
-rw-r--r--   3 ecapriolo supergroup      81743 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/jettison-1.3.4.jar
-rw-r--r--   3 ecapriolo supergroup     533455 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/protobuf-java-2.5.0.jar
-rw-r--r--   3 ecapriolo supergroup     995968 2015-01-16 23:00
/apps/tez-0.4.1-incubating/lib/snappy-java-1.0.4.1.jar
-rw-r--r--   3 ecapriolo supergroup     752332 2015-01-16 23:00
/apps/tez-0.4.1-incubating/tez-api-0.4.1-incubating.jar
-rw-r--r--   3 ecapriolo supergroup      34089 2015-01-16 23:00
/apps/tez-0.4.1-incubating/tez-common-0.4.1-incubating.jar
-rw-r--r--   3 ecapriolo supergroup     980132 2015-01-16 23:00
/apps/tez-0.4.1-incubating/tez-dag-0.4.1-incubating.jar
-rw-r--r--   3 ecapriolo supergroup     246395 2015-01-16 23:00
/apps/tez-0.4.1-incubating/tez-mapreduce-0.4.1-incubating.jar
-rw-r--r--   3 ecapriolo supergroup     199984 2015-01-16 23:00
/apps/tez-0.4.1-incubating/tez-mapreduce-examples-0.4.1-incubating.jar
-rw-r--r--   3 ecapriolo supergroup     114676 2015-01-16 23:00
/apps/tez-0.4.1-incubating/tez-runtime-internals-0.4.1-incubating.jar
-rw-r--r--   3 ecapriolo supergroup     352835 2015-01-16 23:00
/apps/tez-0.4.1-incubating/tez-runtime-library-0.4.1-incubating.jar
-rw-r--r--   3 ecapriolo supergroup       6832 2015-01-16 23:00
/apps/tez-0.4.1-incubating/tez-tests-0.4.1-incubating.jar

This is my tez-site.xml

<configuration>
  <property>
    <name>tez.lib.uris</name>
    <value>${fs.default.name}/apps/tez-0.4.1-incubating,${fs.default.name
}/apps/tez-0.4.1-incubating/lib/</value>
  </property>
</configuration>

[ecapriolo@production-hadoop-cdh-69-7 ~]$ ls -lahR
/home/ecapriolo/tez-0.4.1-incubating/
/home/ecapriolo/tez-0.4.1-incubating/:
total 2.7M
drwxrwxr-x 3 ecapriolo ecapriolo 4.0K Jan 16 22:54 .
drwx------ 7 ecapriolo ecapriolo  20K Jan 20 15:20 ..
drwxrwxr-x 2 ecapriolo ecapriolo 4.0K Jan 16 22:54 lib
-rw-rw-r-- 1 ecapriolo ecapriolo 735K Jan 16 22:54
tez-api-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapriolo ecapriolo  34K Jan 16 22:54
tez-common-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 958K Jan 16 22:54
tez-dag-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 241K Jan 16 22:54
tez-mapreduce-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 196K Jan 16 22:54
tez-mapreduce-examples-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 112K Jan 16 22:54
tez-runtime-internals-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 345K Jan 16 22:54
tez-runtime-library-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 6.7K Jan 16 22:54
tez-tests-0.4.1-incubating.jar

/home/ecapriolo/tez-0.4.1-incubating/lib:
total 6.8M
drwxrwxr-x 2 ecapriolo ecapriolo 4.0K Jan 16 22:54 .
drwxrwxr-x 3 ecapriolo ecapriolo 4.0K Jan 16 22:54 ..
-rw-rw-r-- 1 ecapriolo ecapriolo 297K Jan 16 22:54 avro-1.7.4.jar
-rw-rw-r-- 1 ecapriolo ecapriolo  41K Jan 16 22:54 commons-cli-1.2.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 596K Jan 16 22:54
commons-collections4-4.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 1.6M Jan 16 22:54 guava-11.0.2.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 694K Jan 16 22:54 guice-3.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 641K Jan 16 22:54
hadoop-mapreduce-client-common-2.2.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 1.4M Jan 16 22:54
hadoop-mapreduce-client-core-2.2.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo  22K Jan 16 22:54
hadoop-mapreduce-client-shuffle-2.2.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo  80K Jan 16 22:54 jettison-1.3.4.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 521K Jan 16 22:54 protobuf-java-2.5.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 973K Jan 16 22:54 snappy-java-1.0.4.1.jar

tez.sh
TEZ_CONF_DIR=/home/ecapriolo
TEZ_JARS=/home/ecapriolo/tez-0.4.1-incubating
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
#hive -hiveconf mapreduce.framework.name=yarn-tez
#hive -hiveconf hive.root.logger=DEGUG,console
hive

hive> set
hive.execution.engine=tez;
hive> select sum(viral_count) from author_article_hourly where
dt=2015011622;
Total jobs = 1
Launching Job 1 out of 1


Status: Running (application id: application_1420748315294_70716)

Map 1: -/-    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Map 1: 0/1    Reducer 2: 0/1
Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1420748315294_70716_1_01,
diagnostics=[Task failed, taskId=task_1420748315294_70716_1_01_000000,
diagnostics=[AttemptID:attempt_1420748315294_70716_1_01_000000_0
Info:Container container_1420748315294_70716_01_000002 COMPLETED with
diagnostics set to [Exception from container-launch.
Container id: container_1420748315294_70716_01_000002
Exit code: 255
Stack trace: ExitCodeException exitCode=255:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
    at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
    at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
    at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Other logs are here (from the containers that I can extract from yawn.

Exit code: 255
Stack trace: ExitCodeException exitCode=255:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
    at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
    at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
    at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

application_1420748315294_70716

digraph _1 {
graph [ label="_1", fontsize=24, fontname=Helvetica];
node [fontsize=12, fontname=Helvetica];
edge [fontsize=9, fontcolor=blue, fontname=Arial];
"_1.Map_1_author_article_hourly" [ label = "Map_1[author_article_hourly]",
shape = "box" ];
"_1.Map_1_author_article_hourly" -> "_1.Map_1" [ label = "Input
[inputClass=MRInputLegacy,\n initializer=MRInputAMSplitGenerator]" ];
"_1.Reducer_2" [ label = "Reducer_2[ReduceTezProcessor]" ];
"_1.Reducer_2" -> "_1.Reducer_2_out_Reducer_2" [ label = "Output
[outputClass=MROutput,\n initializer=]" ];
"_1.Map_1" [ label = "Map_1[MapTezProcessor]" ];
"_1.Map_1" -> "_1.Reducer_2" [ label = "[input=OnFileSortedOutput,\n
output=ShuffledMergedInputLegacy,\n dataMovement=SCATTER_GATHER,\n
schedulingType=SEQUENTIAL]" ];
"_1.Reducer_2_out_Reducer_2" [ label = "Reducer_2[out_Reducer_2]", shape =
"box" ];
}


Container exited with a non-zero exit code 255
]], Vertex failed as one or more tasks failed. failedTasks:1]
Vertex killed, vertexName=Reducer 2,
vertexId=vertex_1420748315294_70716_1_00, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
DAG failed due to vertex failure. failedVertices:1 killedVertices:1,
counters=Counters: 2, org.apache.tez.common.counters.DAGCounter,
NUM_FAILED_TASKS=4, TOTAL_LAUNCHED_TASKS=4
2015-01-20 15:02:21,934 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.DAGImpl: DAG: dag_1420748315294_70716_1
finished with state: FAILED
2015-01-20 15:02:21,934 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.DAGImpl: dag_1420748315294_70716_1
transitioned from TERMINATING to FAILED
2015-01-20 15:02:21,935 INFO [AsyncDispatcher event handler]
org.apache.tez.dag.app.DAGAppMaster: DAG completed,
dagId=dag_1420748315294_70716_1, dagState=FAILED
2015-01-20 15:02:21,935 INFO [AsyncDispatcher event handler]
org.apache.tez.common.TezUtils: Redirecting log files based on addend:
dag_1420748315294_70716_1_post

Has anyone got this working or have and ideas as to what is up here?

Thanks,
Edward

Mime
View raw message