Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50D9A176D1 for ; Tue, 28 Oct 2014 20:07:34 +0000 (UTC) Received: (qmail 12458 invoked by uid 500); 28 Oct 2014 20:07:34 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 12428 invoked by uid 500); 28 Oct 2014 20:07:34 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 12402 invoked by uid 99); 28 Oct 2014 20:07:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Oct 2014 20:07:34 +0000 Date: Tue, 28 Oct 2014 20:07:34 +0000 (UTC) From: "jun aoki (JIRA)" To: dev@ambari.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (AMBARI-7994) Query in Tez throws RuntimeException due to buildSupportsSnappy MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMBARI-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187368#comment-14187368 ] jun aoki commented on AMBARI-7994: ---------------------------------- Tested the patch and it works well. {code} hive> select name FROM school order by name asc limit 10; Query ID = hive_20141028200505_43e02f5f-fec3-4771-8644-426a580aec28 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1414523459027_0004) Map 1: -/- Reducer 2: 0/1 Map 1: 0/1 Reducer 2: 0/1 Map 1: 0/1 Reducer 2: 0/1 Map 1: 0(+1)/1 Reducer 2: 0/1 Map 1: 1/1 Reducer 2: 0(+1)/1 Map 1: 1/1 Reducer 2: 1/1 Status: Finished successfully in 7.15 seconds OK "Abilene Christian University" "Adelphi University" "Adrian College" "Alabama A&M University" "Alabama State University" "Albany State University" "Albertson College of Idaho" "Albion College" "Albright College" "Alcorn State University" Time taken: 10.824 seconds, Fetched: 10 row(s) hive> {code} I just have a question to [~afernandez] on the reviewboard due to lack of my knowledge. > Query in Tez throws RuntimeException due to buildSupportsSnappy > --------------------------------------------------------------- > > Key: AMBARI-7994 > URL: https://issues.apache.org/jira/browse/AMBARI-7994 > Project: Ambari > Issue Type: Bug > Components: ambari-server > Affects Versions: 1.7.0 > Reporter: Alejandro Fernandez > Assignee: Alejandro Fernandez > Priority: Blocker > Fix For: 1.7.0 > > Attachments: AMBARI-7994.patch > > > Deployed a cluster with Ambari including HDFS, YARN, Tez, Hive, Oozie, Zookeeper. > Ambari server version is 2.0.0-224 > HDP version is 2.2.0.0-1014 > Changed hive.execution.engine from mr to tez, and then ran the following, > {code} > $ su - hive > $ cd /tmp > $ wget http://seanlahman.com/files/database/lahman591-csv.zip > $ unzip lahman591-csv.zip > $hdfs dfs -copyFromLocal Schools.csv /tmp > $ hive > > CREATE TABLE school (id STRING, name STRING, city STRING, state STRING, nick STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\054' STORED AS TEXTFILE; > > LOAD DATA LOCAL INPATH '/tmp/Schools.csv' INTO TABLE school; > {code} > {code} > hive> select name FROM school order by name asc limit 10; > Query ID = hive_20141023223030_f687e5b7-8050-41b1-9dfd-05d0f6d0b22f > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id application_1414101936665_0007) > Map 1: -/- Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0(+1)/1 Reducer 2: 0/1 > Map 1: 0(+1)/1 Reducer 2: 0/1 > Map 1: 0(+1)/1 Reducer 2: 0/1 > Map 1: 0(+1)/1 Reducer 2: 0/1 > Map 1: 0(+1)/1 Reducer 2: 0/1 > Map 1: 0(+1)/1 Reducer 2: 0/1 > Map 1: 0(+1)/1 Reducer 2: 0/1 > Status: Failed > Vertex failed, vertexName=Map 1, vertexId=vertex_1414101936665_0007_1_00, diagnostics=[Task failed, taskId=task_1414101936665_0007_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142) > at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) > at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) > at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:133) > at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:191) > at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.(DefaultSorter.java:119) > at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:114) > at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:145) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) > ... 13 more > ], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142) > at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) > at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) > at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:133) > at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:191) > at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.(DefaultSorter.java:119) > at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:114) > at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:145) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) > ... 13 more > ], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142) > at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) > at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) > at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:133) > at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:191) > at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.(DefaultSorter.java:119) > at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:114) > at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:145) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) > ... 13 more > ], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142) > at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) > at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) > at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) > at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:133) > at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:191) > at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.(DefaultSorter.java:119) > at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:114) > at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:145) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) > ... 13 more > ]], Vertex failed as one or more tasks failed. failedTasks:1] > Vertex killed, vertexName=Reducer 2, vertexId=vertex_1414101936665_0007_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as other vertex failed. failedTasks:0] > DAG failed due to vertex failure. failedVertices:1 killedVertices:1 > FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)