Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 36FB5177C2 for ; Tue, 20 Jan 2015 19:02:34 +0000 (UTC) Received: (qmail 4462 invoked by uid 500); 20 Jan 2015 19:02:31 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 4397 invoked by uid 500); 20 Jan 2015 19:02:31 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 4387 invoked by uid 99); 20 Jan 2015 19:02:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jan 2015 19:02:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of pjayachandran@hortonworks.com designates 209.85.160.172 as permitted sender) Received: from [209.85.160.172] (HELO mail-yk0-f172.google.com) (209.85.160.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jan 2015 19:02:05 +0000 Received: by mail-yk0-f172.google.com with SMTP id q9so7660724ykb.3 for ; Tue, 20 Jan 2015 11:02:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=uoYtS7krHqEl3LrjxENjywYqK2ZVdQG9STv8VSq8RrE=; b=gQgUxvTyWMMnYHQMdiTvp+zP16+DHAe024dHvPKJZRoMv2qnrL6VQ1zwDjotZvU7Ve lyOtbjijjmOW+FPKRoXtFms85+6ETCZWnnPWpuYa2e2tfOLq2ZLAyBBBzh3t680bFptV 5bMVcCdZGY8BbLAtqNYuZlNEApybt2O3z+rV+9tUU8zokohHOHHIm93EKRVHvP97vsl4 dCcJy1lpXFuw0lTj4zdfrAfdohkJHV88ehlweELvR5K063q1RHbe4w28za5UGPEWKXVm yy3QNbled53o1Tw/MPxlUpCWvO0cgOoFa5EGSj7CuBNmXDS43IBwZ/CAUxldk8rhMofL LURQ== X-Gm-Message-State: ALoCoQlN/tLJyBqIEbb6zuya0XfvPi+lr/UQoBSuXnJanqH9KQ+aQ4aqLnejnioaEHxq2OFXrCWhk5Pp16+l3TpA/a56mwynYLKJ9IEnPkYuK0sPA4Xlw6E= MIME-Version: 1.0 X-Received: by 10.236.31.2 with SMTP id l2mr965052yha.2.1421780523141; Tue, 20 Jan 2015 11:02:03 -0800 (PST) Received: by 10.170.225.136 with HTTP; Tue, 20 Jan 2015 11:02:03 -0800 (PST) In-Reply-To: References: Date: Tue, 20 Jan 2015 11:02:03 -0800 Message-ID: Subject: Re: Getting Tez working against cdh 5.3 From: Prasanth Jayachandran To: "user@hive.apache.org" Content-Type: multipart/alternative; boundary=089e0118250c07e626050d1a139e X-Virus-Checked: Checked by ClamAV on apache.org --089e0118250c07e626050d1a139e Content-Type: text/plain; charset=UTF-8 My guess is.. "java" binary is not in PATH of the shell script that launches the container.. try creating a symbolic link in /bin/ to point to java.. On Tue, Jan 20, 2015 at 7:22 AM, Edward Capriolo wrote: > It seems that CDH does not ship with enough jars to run tez out of the box. > > I have found the related cloudera forked pom. > > In this pom hive is built against tez 0.4.1-incubating-tez2.0-SHAPSHOT > > Thus I followed the instructions here: > > http://tez.apache.org/install_pre_0_5_0.html > > hive> dfs -lsr /apps > > ; > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - ecapriolo supergroup 0 2015-01-16 23:00 > /apps/tez-0.4.1-incubating > drwxr-xr-x - ecapriolo supergroup 0 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib > -rw-r--r-- 3 ecapriolo supergroup 303139 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/avro-1.7.4.jar > -rw-r--r-- 3 ecapriolo supergroup 41123 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/commons-cli-1.2.jar > -rw-r--r-- 3 ecapriolo supergroup 610259 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/commons-collections4-4.0.jar > -rw-r--r-- 3 ecapriolo supergroup 1648200 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/guava-11.0.2.jar > -rw-r--r-- 3 ecapriolo supergroup 710492 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/guice-3.0.jar > -rw-r--r-- 3 ecapriolo supergroup 656365 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-common-2.2.0.jar > -rw-r--r-- 3 ecapriolo supergroup 1455001 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-core-2.2.0.jar > -rw-r--r-- 3 ecapriolo supergroup 21537 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-shuffle-2.2.0.jar > -rw-r--r-- 3 ecapriolo supergroup 81743 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/jettison-1.3.4.jar > -rw-r--r-- 3 ecapriolo supergroup 533455 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/protobuf-java-2.5.0.jar > -rw-r--r-- 3 ecapriolo supergroup 995968 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/lib/snappy-java-1.0.4.1.jar > -rw-r--r-- 3 ecapriolo supergroup 752332 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/tez-api-0.4.1-incubating.jar > -rw-r--r-- 3 ecapriolo supergroup 34089 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/tez-common-0.4.1-incubating.jar > -rw-r--r-- 3 ecapriolo supergroup 980132 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/tez-dag-0.4.1-incubating.jar > -rw-r--r-- 3 ecapriolo supergroup 246395 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/tez-mapreduce-0.4.1-incubating.jar > -rw-r--r-- 3 ecapriolo supergroup 199984 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/tez-mapreduce-examples-0.4.1-incubating.jar > -rw-r--r-- 3 ecapriolo supergroup 114676 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/tez-runtime-internals-0.4.1-incubating.jar > -rw-r--r-- 3 ecapriolo supergroup 352835 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/tez-runtime-library-0.4.1-incubating.jar > -rw-r--r-- 3 ecapriolo supergroup 6832 2015-01-16 23:00 > /apps/tez-0.4.1-incubating/tez-tests-0.4.1-incubating.jar > > This is my tez-site.xml > > > > tez.lib.uris > ${fs.default.name}/apps/tez-0.4.1-incubating,${fs.default.name > }/apps/tez-0.4.1-incubating/lib/ > > > > [ecapriolo@production-hadoop-cdh-69-7 ~]$ ls -lahR > /home/ecapriolo/tez-0.4.1-incubating/ > /home/ecapriolo/tez-0.4.1-incubating/: > total 2.7M > drwxrwxr-x 3 ecapriolo ecapriolo 4.0K Jan 16 22:54 . > drwx------ 7 ecapriolo ecapriolo 20K Jan 20 15:20 .. > drwxrwxr-x 2 ecapriolo ecapriolo 4.0K Jan 16 22:54 lib > -rw-rw-r-- 1 ecapriolo ecapriolo 735K Jan 16 22:54 > tez-api-0.4.1-incubating.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 34K Jan 16 22:54 > tez-common-0.4.1-incubating.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 958K Jan 16 22:54 > tez-dag-0.4.1-incubating.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 241K Jan 16 22:54 > tez-mapreduce-0.4.1-incubating.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 196K Jan 16 22:54 > tez-mapreduce-examples-0.4.1-incubating.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 112K Jan 16 22:54 > tez-runtime-internals-0.4.1-incubating.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 345K Jan 16 22:54 > tez-runtime-library-0.4.1-incubating.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 6.7K Jan 16 22:54 > tez-tests-0.4.1-incubating.jar > > /home/ecapriolo/tez-0.4.1-incubating/lib: > total 6.8M > drwxrwxr-x 2 ecapriolo ecapriolo 4.0K Jan 16 22:54 . > drwxrwxr-x 3 ecapriolo ecapriolo 4.0K Jan 16 22:54 .. > -rw-rw-r-- 1 ecapriolo ecapriolo 297K Jan 16 22:54 avro-1.7.4.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 41K Jan 16 22:54 commons-cli-1.2.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 596K Jan 16 22:54 > commons-collections4-4.0.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 1.6M Jan 16 22:54 guava-11.0.2.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 694K Jan 16 22:54 guice-3.0.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 641K Jan 16 22:54 > hadoop-mapreduce-client-common-2.2.0.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 1.4M Jan 16 22:54 > hadoop-mapreduce-client-core-2.2.0.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 22K Jan 16 22:54 > hadoop-mapreduce-client-shuffle-2.2.0.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 80K Jan 16 22:54 jettison-1.3.4.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 521K Jan 16 22:54 protobuf-java-2.5.0.jar > -rw-rw-r-- 1 ecapriolo ecapriolo 973K Jan 16 22:54 snappy-java-1.0.4.1.jar > > tez.sh > TEZ_CONF_DIR=/home/ecapriolo > TEZ_JARS=/home/ecapriolo/tez-0.4.1-incubating > export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/* > #hive -hiveconf mapreduce.framework.name=yarn-tez > #hive -hiveconf hive.root.logger=DEGUG,console > hive > > hive> set > hive.execution.engine=tez; > hive> select sum(viral_count) from author_article_hourly where dt= > 2015011622; > Total jobs = 1 > Launching Job 1 out of 1 > > > Status: Running (application id: application_1420748315294_70716) > > Map 1: -/- Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Status: Failed > Vertex failed, vertexName=Map 1, vertexId=vertex_1420748315294_70716_1_01, > diagnostics=[Task failed, taskId=task_1420748315294_70716_1_01_000000, > diagnostics=[AttemptID:attempt_1420748315294_70716_1_01_000000_0 > Info:Container container_1420748315294_70716_01_000002 COMPLETED with > diagnostics set to [Exception from container-launch. > Container id: container_1420748315294_70716_01_000002 > Exit code: 255 > Stack trace: ExitCodeException exitCode=255: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > Other logs are here (from the containers that I can extract from yawn. > > Exit code: 255 > Stack trace: ExitCodeException exitCode=255: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > application_1420748315294_70716 > > digraph _1 { > graph [ label="_1", fontsize=24, fontname=Helvetica]; > node [fontsize=12, fontname=Helvetica]; > edge [fontsize=9, fontcolor=blue, fontname=Arial]; > "_1.Map_1_author_article_hourly" [ label = "Map_1[author_article_hourly]", > shape = "box" ]; > "_1.Map_1_author_article_hourly" -> "_1.Map_1" [ label = "Input > [inputClass=MRInputLegacy,\n initializer=MRInputAMSplitGenerator]" ]; > "_1.Reducer_2" [ label = "Reducer_2[ReduceTezProcessor]" ]; > "_1.Reducer_2" -> "_1.Reducer_2_out_Reducer_2" [ label = "Output > [outputClass=MROutput,\n initializer=]" ]; > "_1.Map_1" [ label = "Map_1[MapTezProcessor]" ]; > "_1.Map_1" -> "_1.Reducer_2" [ label = "[input=OnFileSortedOutput,\n > output=ShuffledMergedInputLegacy,\n dataMovement=SCATTER_GATHER,\n > schedulingType=SEQUENTIAL]" ]; > "_1.Reducer_2_out_Reducer_2" [ label = "Reducer_2[out_Reducer_2]", shape = > "box" ]; > } > > > Container exited with a non-zero exit code 255 > ]], Vertex failed as one or more tasks failed. failedTasks:1] > Vertex killed, vertexName=Reducer 2, > vertexId=vertex_1420748315294_70716_1_00, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex killed as other vertex failed. > failedTasks:0] > DAG failed due to vertex failure. failedVertices:1 killedVertices:1, > counters=Counters: 2, org.apache.tez.common.counters.DAGCounter, > NUM_FAILED_TASKS=4, TOTAL_LAUNCHED_TASKS=4 > 2015-01-20 15:02:21,934 INFO [AsyncDispatcher event handler] > org.apache.tez.dag.app.dag.impl.DAGImpl: DAG: dag_1420748315294_70716_1 > finished with state: FAILED > 2015-01-20 15:02:21,934 INFO [AsyncDispatcher event handler] > org.apache.tez.dag.app.dag.impl.DAGImpl: dag_1420748315294_70716_1 > transitioned from TERMINATING to FAILED > 2015-01-20 15:02:21,935 INFO [AsyncDispatcher event handler] > org.apache.tez.dag.app.DAGAppMaster: DAG completed, > dagId=dag_1420748315294_70716_1, dagState=FAILED > 2015-01-20 15:02:21,935 INFO [AsyncDispatcher event handler] > org.apache.tez.common.TezUtils: Redirecting log files based on addend: > dag_1420748315294_70716_1_post > > Has anyone got this working or have and ideas as to what is up here? > > Thanks, > Edward > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --089e0118250c07e626050d1a139e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
My guess is..
=C2=A0"java" binary is not in = PATH of the shell script that launches the container.. try creating a symbo= lic link in /bin/ to point to java..=C2=A0

On Tue, Jan 20, 2015 at 7:22 AM, Edwar= d Capriolo <edlinuxguru@gmail.com> wrote:
It seems that CDH does not ship with enough jars to run tez out of the bo= x.

I have found the related cloudera forked pom.

= In this pom hive is built against tez 0.4.1-incubating-tez2.0-SHAPSHOT
<= br>
Thus I followed the instructions here:

http://tez.apache.or= g/install_pre_0_5_0.html

hive> dfs -lsr /apps
=C2=A0=C2=A0= =C2=A0 > ;
lsr: DEPRECATED: Please use 'ls -R' instead.
dr= wxr-xr-x=C2=A0=C2=A0 - ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0 2015-01-16 23:00 /apps/tez-0.4.1-incubating
d= rwxr-xr-x=C2=A0=C2=A0 - ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0 2015-01-16 23:00 /apps/tez-0.4.1-incubating/lib<= br>-rw-r--r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 30= 3139 2015-01-16 23:00 /apps/tez-0.4.1-incubating/lib/avro-1.7.4.jar
-rw-= r--r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 411= 23 2015-01-16 23:00 /apps/tez-0.4.1-incubating/lib/commons-cli-1.2.jar
-= rw-r--r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 610259= 2015-01-16 23:00 /apps/tez-0.4.1-incubating/lib/commons-collections4-4.0.j= ar
-rw-r--r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0 16482= 00 2015-01-16 23:00 /apps/tez-0.4.1-incubating/lib/guava-11.0.2.jar
-rw-= r--r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 710492 20= 15-01-16 23:00 /apps/tez-0.4.1-incubating/lib/guice-3.0.jar
-rw-r--r--= =C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 656365 2015-01-= 16 23:00 /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-client-common-2.2.= 0.jar
-rw-r--r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0 14= 55001 2015-01-16 23:00 /apps/tez-0.4.1-incubating/lib/hadoop-mapreduce-clie= nt-core-2.2.0.jar
-rw-r--r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 21537 2015-01-16 23:00 /apps/tez-0.4.1-incubating/lib= /hadoop-mapreduce-client-shuffle-2.2.0.jar
-rw-r--r--=C2=A0=C2=A0 3 ecap= riolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 81743 2015-01-16 23:00 /apps= /tez-0.4.1-incubating/lib/jettison-1.3.4.jar
-rw-r--r--=C2=A0=C2=A0 3 ec= apriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 533455 2015-01-16 23:00 /apps/te= z-0.4.1-incubating/lib/protobuf-java-2.5.0.jar
-rw-r--r--=C2=A0=C2=A0 3 = ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 995968 2015-01-16 23:00 /apps/= tez-0.4.1-incubating/lib/snappy-java-1.0.4.1.jar
-rw-r--r--=C2=A0=C2=A0 = 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 752332 2015-01-16 23:00 /app= s/tez-0.4.1-incubating/tez-api-0.4.1-incubating.jar
-rw-r--r--=C2=A0=C2= =A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 34089 2015-01-16 2= 3:00 /apps/tez-0.4.1-incubating/tez-common-0.4.1-incubating.jar
-rw-r--r= --=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 980132 2015-0= 1-16 23:00 /apps/tez-0.4.1-incubating/tez-dag-0.4.1-incubating.jar
-rw-r= --r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 246395 201= 5-01-16 23:00 /apps/tez-0.4.1-incubating/tez-mapreduce-0.4.1-incubating.jar=
-rw-r--r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0 1= 99984 2015-01-16 23:00 /apps/tez-0.4.1-incubating/tez-mapreduce-examples-0.= 4.1-incubating.jar
-rw-r--r--=C2=A0=C2=A0 3 ecapriolo supergroup=C2=A0= =C2=A0=C2=A0=C2=A0 114676 2015-01-16 23:00 /apps/tez-0.4.1-incubating/tez-r= untime-internals-0.4.1-incubating.jar
-rw-r--r--=C2=A0=C2=A0 3 ecapriolo= supergroup=C2=A0=C2=A0=C2=A0=C2=A0 352835 2015-01-16 23:00 /apps/tez-0.4.1= -incubating/tez-runtime-library-0.4.1-incubating.jar
-rw-r--r--=C2=A0=C2= =A0 3 ecapriolo supergroup=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 6832 2015-01= -16 23:00 /apps/tez-0.4.1-incubating/tez-tests-0.4.1-incubating.jar

=
This is my tez-site.xml

<configuration>
=C2=A0 <pr= operty>
=C2=A0=C2=A0=C2=A0 <name>tez.lib.uris</name>
= =C2=A0=C2=A0=C2=A0 <value>${fs.default.name}/apps/tez-0.4.1-incubating,${fs.default.name}/apps/tez-0.4.1-= incubating/lib/</value>
=C2=A0 </property>
</configura= tion>

[ecapriolo@production-hadoop-cdh-69-7 ~]$ ls -lahR /home/ec= apriolo/tez-0.4.1-incubating/
/home/ecapriolo/tez-0.4.1-incubating/:
= total 2.7M
drwxrwxr-x 3 ecapriolo ecapriolo 4.0K Jan 16 22:54 .
drwx-= ----- 7 ecapriolo ecapriolo=C2=A0 20K Jan 20 15:20 ..
drwxrwxr-x 2 ecapr= iolo ecapriolo 4.0K Jan 16 22:54 lib
-rw-rw-r-- 1 ecapriolo ecapriolo 73= 5K Jan 16 22:54 tez-api-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapriolo ecap= riolo=C2=A0 34K Jan 16 22:54 tez-common-0.4.1-incubating.jar
-rw-rw-r-- = 1 ecapriolo ecapriolo 958K Jan 16 22:54 tez-dag-0.4.1-incubating.jar
-rw= -rw-r-- 1 ecapriolo ecapriolo 241K Jan 16 22:54 tez-mapreduce-0.4.1-incubat= ing.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 196K Jan 16 22:54 tez-mapreduce= -examples-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 112K Jan= 16 22:54 tez-runtime-internals-0.4.1-incubating.jar
-rw-rw-r-- 1 ecapri= olo ecapriolo 345K Jan 16 22:54 tez-runtime-library-0.4.1-incubating.jar-rw-rw-r-- 1 ecapriolo ecapriolo 6.7K Jan 16 22:54 tez-tests-0.4.1-incubat= ing.jar

/home/ecapriolo/tez-0.4.1-incubating/lib:
total 6.8M
d= rwxrwxr-x 2 ecapriolo ecapriolo 4.0K Jan 16 22:54 .
drwxrwxr-x 3 ecaprio= lo ecapriolo 4.0K Jan 16 22:54 ..
-rw-rw-r-- 1 ecapriolo ecapriolo 297K = Jan 16 22:54 avro-1.7.4.jar
-rw-rw-r-- 1 ecapriolo ecapriolo=C2=A0 41K J= an 16 22:54 commons-cli-1.2.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 596K Ja= n 16 22:54 commons-collections4-4.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo= 1.6M Jan 16 22:54 guava-11.0.2.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 694= K Jan 16 22:54 guice-3.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 641K Jan 1= 6 22:54 hadoop-mapreduce-client-common-2.2.0.jar
-rw-rw-r-- 1 ecapriolo = ecapriolo 1.4M Jan 16 22:54 hadoop-mapreduce-client-core-2.2.0.jar
-rw-r= w-r-- 1 ecapriolo ecapriolo=C2=A0 22K Jan 16 22:54 hadoop-mapreduce-client-= shuffle-2.2.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo=C2=A0 80K Jan 16 22:5= 4 jettison-1.3.4.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 521K Jan 16 22:54 = protobuf-java-2.5.0.jar
-rw-rw-r-- 1 ecapriolo ecapriolo 973K Jan 16 22:= 54 snappy-java-1.0.4.1.jar

tez.sh
TEZ_CONF_DIR=3D/home/ecap= riolo
TEZ_JARS=3D/home/ecapriolo/tez-0.4.1-incubating
export HADOOP_C= LASSPATH=3D${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
#hive -hiveco= nf mapreduce.= framework.name=3Dyarn-tez
#hive -hiveconf hive.root.logger=3DDEGUG,c= onsole
hive

hive> set hive.execution.engine=3Dtez;=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0
hive> select sum(viral_count) from author_article_hourly where d= t=3D201= 5011622;
Total jobs =3D 1
Launching Job 1 out of 1


Sta= tus: Running (application id: application_1420748315294_70716)

Map 1= : -/-=C2=A0=C2=A0=C2=A0 Reducer 2: 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1=C2= =A0=C2=A0=C2=A0 Reducer 2: 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2=A0= =C2=A0 Reducer 2: 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0 Re= ducer 2: 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0 Reducer 2: = 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0 Reducer 2: 0/1=C2=A0= =C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0 Reducer 2: 0/1=C2=A0=C2=A0=C2= =A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0 Reducer 2: 0/1=C2=A0=C2=A0=C2=A0
M= ap 1: 0/1=C2=A0=C2=A0=C2=A0 Reducer 2: 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1= =C2=A0=C2=A0=C2=A0 Reducer 2: 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2= =A0=C2=A0 Reducer 2: 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0= Reducer 2: 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0 Reducer = 2: 0/1=C2=A0=C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0 Reducer 2: 0/1=C2= =A0=C2=A0=C2=A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0 Reducer 2: 0/1=C2=A0=C2=A0= =C2=A0
Map 1: 0/1=C2=A0=C2=A0=C2=A0 Reducer 2: 0/1=C2=A0=C2=A0=C2=A0 Status: Failed
Vertex failed, vertexName=3DMap 1, vertexId=3Dvertex_14= 20748315294_70716_1_01, diagnostics=3D[Task failed, taskId=3Dtask_142074831= 5294_70716_1_01_000000, diagnostics=3D[AttemptID:attempt_1420748315294_7071= 6_1_01_000000_0 Info:Container container_1420748315294_70716_01_000002 COMP= LETED with diagnostics set to [Exception from container-launch.
Containe= r id: container_1420748315294_70716_01_000002
Exit code: 255
Stack tr= ace: ExitCodeException exitCode=3D255:
=C2=A0=C2=A0=C2=A0 at org.apache= .hadoop.util.Shell.runCommand(Shell.java:538)
=C2=A0=C2=A0=C2=A0 at org.= apache.hadoop.util.Shell.run(Shell.java:455)
=C2=A0=C2=A0=C2=A0 at org.a= pache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
=C2= =A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.server.nodemanager.DefaultContain= erExecutor.launchContainer(DefaultContainerExecutor.java:197)
=C2=A0=C2= =A0=C2=A0 at org.apache.hadoop.yarn.server.nodemanager.containermanager.lau= ncher.ContainerLaunch.call(ContainerLaunch.java:299)
=C2=A0=C2=A0=C2=A0 = at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:81)
=C2=A0=C2=A0=C2=A0 at java.uti= l.concurrent.FutureTask.run(FutureTask.java:262)
=C2=A0=C2=A0=C2=A0 at j= ava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11= 45)
=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Worker= .run(ThreadPoolExecutor.java:615)
=C2=A0=C2=A0=C2=A0 at java.lang.Thread= .run(Thread.java:745)

Other logs are here (from the containers= that I can extract from yawn.

Exit code: 255
Stack trace: ExitCo= deException exitCode=3D255:
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.uti= l.Shell.runCommand(Shell.java:538)
=C2=A0=C2=A0=C2=A0 at org.apache.hado= op.util.Shell.run(Shell.java:455)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoo= p.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
=C2=A0=C2=A0= =C2=A0 at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecuto= r.launchContainer(DefaultContainerExecutor.java:197)
=C2=A0=C2=A0=C2=A0 = at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:299)
=C2=A0=C2=A0=C2=A0 at org.apa= che.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunc= h.call(ContainerLaunch.java:81)
=C2=A0=C2=A0=C2=A0 at java.util.concurre= nt.FutureTask.run(FutureTask.java:262)
=C2=A0=C2=A0=C2=A0 at java.util.c= oncurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
=C2= =A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa= dPoolExecutor.java:615)
=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Threa= d.java:744)

application_1420748315294_70716

digraph _1 {
g= raph [ label=3D"_1", fontsize=3D24, fontname=3DHelvetica];
nod= e [fontsize=3D12, fontname=3DHelvetica];
edge [fontsize=3D9, fontcolor= =3Dblue, fontname=3DArial];
"_1.Map_1_author_article_hourly" [= label =3D "Map_1[author_article_hourly]", shape =3D "box&qu= ot; ];
"_1.Map_1_author_article_hourly" -> "_1.Map_1&q= uot; [ label =3D "Input [inputClass=3DMRInputLegacy,\n initializer=3DM= RInputAMSplitGenerator]" ];
"_1.Reducer_2" [ label =3D &q= uot;Reducer_2[ReduceTezProcessor]" ];
"_1.Reducer_2" ->= ; "_1.Reducer_2_out_Reducer_2" [ label =3D "Output [outputCl= ass=3DMROutput,\n initializer=3D]" ];
"_1.Map_1" [ label = =3D "Map_1[MapTezProcessor]" ];
"_1.Map_1" -> &qu= ot;_1.Reducer_2" [ label =3D "[input=3DOnFileSortedOutput,\n outp= ut=3DShuffledMergedInputLegacy,\n dataMovement=3DSCATTER_GATHER,\n scheduli= ngType=3DSEQUENTIAL]" ];
"_1.Reducer_2_out_Reducer_2" [ l= abel =3D "Reducer_2[out_Reducer_2]", shape =3D "box" ];=
}


Container exited with a non-zero exit code 255
]], Vert= ex failed as one or more tasks failed. failedTasks:1]
Vertex killed, ver= texName=3DReducer 2, vertexId=3Dvertex_1420748315294_70716_1_00, diagnostic= s=3D[Vertex received Kill while in RUNNING state., Vertex killed as other v= ertex failed. failedTasks:0]
DAG failed due to vertex failure. failedVer= tices:1 killedVertices:1, counters=3DCounters: 2, org.apache.tez.common.cou= nters.DAGCounter, NUM_FAILED_TASKS=3D4, TOTAL_LAUNCHED_TASKS=3D4
2015-01= -20 15:02:21,934 INFO [AsyncDispatcher event handler] org.apache.tez.dag.ap= p.dag.impl.DAGImpl: DAG: dag_1420748315294_70716_1 finished with state: FAI= LED
2015-01-20 15:02:21,934 INFO [AsyncDispatcher event handler] org.apa= che.tez.dag.app.dag.impl.DAGImpl: dag_1420748315294_70716_1 transitioned fr= om TERMINATING to FAILED
2015-01-20 15:02:21,935 INFO [AsyncDispatcher e= vent handler] org.apache.tez.dag.app.DAGAppMaster: DAG completed, dagId=3Dd= ag_1420748315294_70716_1, dagState=3DFAILED
2015-01-20 15:02:21,935 INFO= [AsyncDispatcher event handler] org.apache.tez.common.TezUtils: Redirectin= g log files based on addend: dag_1420748315294_70716_1_post

Ha= s anyone got this working or have and ideas as to what is up here?

<= /div>Thanks,
Edward




CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --089e0118250c07e626050d1a139e--