hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11325) Infinite loop in HiveHFileOutputFormat
Date Fri, 16 Oct 2015 10:41:05 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960498#comment-14960498
] 

Hive QA commented on HIVE-11325:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12746327/HIVE-11325.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5674/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5674/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5674/

Messages:
{noformat}
**** This message was trimmed, see log for full details ****
[WARNING] /data/hive-ptest/working/apache-github-source-source/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java:
Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-cli ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-github-source-source/cli/src/test/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-cli ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/cli/target/tmp
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/cli/target/warehouse
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/cli/target/tmp/conf
     [copy] Copying 14 files to /data/hive-ptest/working/apache-github-source-source/cli/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-cli ---
[INFO] Compiling 4 source files to /data/hive-ptest/working/apache-github-source-source/cli/target/test-classes
[WARNING] /data/hive-ptest/working/apache-github-source-source/cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java:
/data/hive-ptest/working/apache-github-source-source/cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java
uses unchecked or unsafe operations.
[WARNING] /data/hive-ptest/working/apache-github-source-source/cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java:
Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-cli ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-cli ---
[INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/cli/target/hive-cli-2.0.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-cli ---
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-cli ---
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/cli/target/hive-cli-2.0.0-SNAPSHOT.jar
to /home/hiveptest/.m2/repository/org/apache/hive/hive-cli/2.0.0-SNAPSHOT/hive-cli-2.0.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/cli/pom.xml to /home/hiveptest/.m2/repository/org/apache/hive/hive-cli/2.0.0-SNAPSHOT/hive-cli-2.0.0-SNAPSHOT.pom
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Contrib 2.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-contrib ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/contrib/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/contrib (includes = [datanucleus.log,
derby.log], excludes = [])
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-contrib ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-contrib ---
[WARNING] Invalid project model for artifact [pentaho-aggdesigner-algorithm:org.pentaho:5.1.5-jhyde].
It will be ignored by the remote resources Mojo.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-contrib ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-github-source-source/contrib/src/main/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-contrib ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-contrib ---
[INFO] Compiling 40 source files to /data/hive-ptest/working/apache-github-source-source/contrib/target/classes
[WARNING] /data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example/UDAFExampleMax.java:
Some input files use or override a deprecated API.
[WARNING] /data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example/UDAFExampleMax.java:
Recompile with -Xlint:deprecation for details.
[WARNING] /data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java:
/data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java
uses unchecked or unsafe operations.
[WARNING] /data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java:
Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-contrib
---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-github-source-source/contrib/src/test/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-contrib ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/contrib/target/tmp
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/contrib/target/warehouse
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/contrib/target/tmp/conf
     [copy] Copying 14 files to /data/hive-ptest/working/apache-github-source-source/contrib/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-contrib ---
[INFO] Compiling 2 source files to /data/hive-ptest/working/apache-github-source-source/contrib/target/test-classes
[WARNING] /data/hive-ptest/working/apache-github-source-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java:
/data/hive-ptest/working/apache-github-source-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java
uses or overrides a deprecated API.
[WARNING] /data/hive-ptest/working/apache-github-source-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java:
Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-contrib ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-contrib ---
[INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/contrib/target/hive-contrib-2.0.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-contrib ---
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-contrib ---
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/contrib/target/hive-contrib-2.0.0-SNAPSHOT.jar
to /home/hiveptest/.m2/repository/org/apache/hive/hive-contrib/2.0.0-SNAPSHOT/hive-contrib-2.0.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/contrib/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/hive-contrib/2.0.0-SNAPSHOT/hive-contrib-2.0.0-SNAPSHOT.pom
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive HBase Handler 2.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hbase-handler ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/hbase-handler/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/hbase-handler (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-hbase-handler
---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-hbase-handler ---
[WARNING] Invalid project model for artifact [pentaho-aggdesigner-algorithm:org.pentaho:5.1.5-jhyde].
It will be ignored by the remote resources Mojo.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-hbase-handler ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-github-source-source/hbase-handler/src/main/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hbase-handler ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hbase-handler ---
[INFO] Compiling 33 source files to /data/hive-ptest/working/apache-github-source-source/hbase-handler/target/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/AbstractHBaseKeyFactory.java:
Some input files use or override a deprecated API.
[WARNING] /data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/AbstractHBaseKeyFactory.java:
Recompile with -Xlint:deprecation for details.
[WARNING] /data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java:
Some input files use unchecked or unsafe operations.
[WARNING] /data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java:
Recompile with -Xlint:unchecked for details.
[INFO] 4 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java:[164,24]
cannot find symbol
  symbol:   method isDir()
  location: variable srcDir of type org.apache.hadoop.fs.Path
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Hive .............................................. SUCCESS [13.189s]
[INFO] Hive Shims Common ................................. SUCCESS [15.029s]
[INFO] Hive Shims 0.20S .................................. SUCCESS [3.959s]
[INFO] Hive Shims 0.23 ................................... SUCCESS [11.769s]
[INFO] Hive Shims Scheduler .............................. SUCCESS [2.347s]
[INFO] Hive Shims ........................................ SUCCESS [3.572s]
[INFO] Hive Storage API .................................. SUCCESS [3.830s]
[INFO] Hive Common ....................................... SUCCESS [22.237s]
[INFO] Hive Serde ........................................ SUCCESS [20.788s]
[INFO] Hive Metastore .................................... SUCCESS [1:20.849s]
[INFO] Hive Ant Utilities ................................ SUCCESS [2.109s]
[INFO] Hive Llap Client .................................. SUCCESS [8.562s]
[INFO] Spark Remote Client ............................... SUCCESS [15.956s]
[INFO] Hive Query Language ............................... SUCCESS [3:23.412s]
[INFO] Hive Service ...................................... SUCCESS [15.057s]
[INFO] Hive Accumulo Handler ............................. SUCCESS [7.041s]
[INFO] Hive JDBC ......................................... SUCCESS [23.719s]
[INFO] Hive Beeline ...................................... SUCCESS [4.057s]
[INFO] Hive CLI .......................................... SUCCESS [5.716s]
[INFO] Hive Contrib ...................................... SUCCESS [2.653s]
[INFO] Hive HBase Handler ................................ FAILURE [3.810s]
[INFO] Hive HCatalog ..................................... SKIPPED
[INFO] Hive HCatalog Core ................................ SKIPPED
[INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
[INFO] Hive HCatalog Server Extensions ................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
[INFO] Hive HCatalog Webhcat ............................. SKIPPED
[INFO] Hive HCatalog Streaming ........................... SKIPPED
[INFO] Hive HPL/SQL ...................................... SKIPPED
[INFO] Hive HWI .......................................... SKIPPED
[INFO] Hive ODBC ......................................... SKIPPED
[INFO] Hive Shims Aggregator ............................. SKIPPED
[INFO] Hive TestUtils .................................... SKIPPED
[INFO] Hive Packaging .................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7:52.943s
[INFO] Finished at: Fri Oct 16 06:40:38 EDT 2015
[INFO] Final Memory: 215M/1065M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile
(default-compile) on project hive-hbase-handler: Compilation failure
[ERROR] /data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java:[164,24]
cannot find symbol
[ERROR] symbol:   method isDir()
[ERROR] location: variable srcDir of type org.apache.hadoop.fs.Path
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following
articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hive-hbase-handler
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12746327 - PreCommit-HIVE-TRUNK-Build

> Infinite loop in HiveHFileOutputFormat
> --------------------------------------
>
>                 Key: HIVE-11325
>                 URL: https://issues.apache.org/jira/browse/HIVE-11325
>             Project: Hive
>          Issue Type: Bug
>          Components: HBase Handler
>    Affects Versions: 1.0.0
>            Reporter: Harsh J
>            Assignee: Harsh J
>         Attachments: HIVE-11325.patch
>
>
> No idea why {{hbase_handler_bulk.q}} does not catch this if its being run regularly in
Hive builds, but here's the gist of the issue:
> The condition at https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java#L152-L164
indicates that we will infinitely loop until we find a file whose last path component (the
name) is equal to the column family name.
> In execution, however, the iteration enters an actual infinite loop cause the file we
end up considering as the srcDir name, is actually the region file, whose name will never
match the family name.
> This is an example of the IPC the listing loop of a 100% progress task gets stuck in:
> {code}
> 2015-07-21 10:32:20,662 TRACE [main] org.apache.hadoop.ipc.ProtobufRpcEngine: 1: Call
-> cdh54.vm/172.16.29.132:8020: getListing {src: "/user/hive/warehouse/hbase_test/_temporary/1/_temporary/attempt_1436935612068_0011_m_000000_0/family/97112ac1c09548ae87bd85af072d2e8c"
startAfter: "" needLocation: false}
> 2015-07-21 10:32:20,662 DEBUG [IPC Parameter Sending Thread #1] org.apache.hadoop.ipc.Client:
IPC Client (1551465414) connection to cdh54.vm/172.16.29.132:8020 from hive sending #510346
> 2015-07-21 10:32:20,662 DEBUG [IPC Client (1551465414) connection to cdh54.vm/172.16.29.132:8020
from hive] org.apache.hadoop.ipc.Client: IPC Client (1551465414) connection to cdh54.vm/172.16.29.132:8020
from hive got value #510346
> 2015-07-21 10:32:20,662 DEBUG [main] org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getListing
took 0ms
> 2015-07-21 10:32:20,662 TRACE [main] org.apache.hadoop.ipc.ProtobufRpcEngine: 1: Response
<- cdh54.vm/172.16.29.132:8020: getListing {dirList { partialListing { fileType: IS_FILE
path: "" length: 863 permission { perm: 4600 } owner: "hive" group: "hive" modification_time:
1437454718130 access_time: 1437454717973 block_replication: 1 blocksize: 134217728 fileId:
33960 childrenNum: 0 storagePolicy: 0 } remainingEntries: 0 }}
> {code}
> The path we are getting out of the listing results is {{/user/hive/warehouse/hbase_test/_temporary/1/_temporary/attempt_1436935612068_0011_m_000000_0/family/97112ac1c09548ae87bd85af072d2e8c}},
but instead of checking the path's parent {{family}} we're instead looping infinitely over
its hashed filename {{97112ac1c09548ae87bd85af072d2e8c}} cause it does not match {{family}}.
> It stays in the infinite loop therefore, until the MR framework kills it away due to
an idle task timeout (and then since the subsequent task attempts fail outright, the job fails).
> While doing a {{getPath().getParent()}} will resolve that, is that infinite loop even
necessary? Especially given the fact that we throw exceptions if there are no entries or there
is more than one entry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message