hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20948) Eliminate file rename in compactor
Date Mon, 24 Feb 2020 18:36:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17043742#comment-17043742
] 

Hive QA commented on HIVE-20948:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12994295/HIVE-20948.06.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20804/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20804/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20804/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ date '+%Y-%m-%d %T.%3N'
2020-02-24 18:34:14.679
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20804/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-02-24 18:34:14.681
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   9202cef..92d9f7d  master     -> origin/master
+ git reset --hard HEAD
HEAD is now at 9202cef HIVE-21543: Use FilterHooks for show compactions (Laszlo Pinter via
Adam Szita)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 92d9f7d HIVE-22923: Extract cumulative cost metadata from HiveRelMdDistinctRowCount
metadata provider (Jesus Camacho Rodriguez, reviewed by Miklos Gergely)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-02-24 18:34:17.500
+ rm -rf ../yetus_PreCommit-HIVE-Build-20804
+ mkdir ../yetus_PreCommit-HIVE-Build-20804
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20804
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20804/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java: does not exist
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java: does not exist
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java: does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java: does not exist
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java: does not
exist in index
Trying to apply the patch with -p1
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java:41
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java:41
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java'
with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-20804
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12994295 - PreCommit-HIVE-Build

> Eliminate file rename in compactor
> ----------------------------------
>
>                 Key: HIVE-20948
>                 URL: https://issues.apache.org/jira/browse/HIVE-20948
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 4.0.0
>            Reporter: Eugene Koifman
>            Assignee: László Pintér
>            Priority: Major
>         Attachments: HIVE-20948.01.patch, HIVE-20948.02.patch, HIVE-20948.03.patch, HIVE-20948.04.patch,
HIVE-20948.05.patch, HIVE-20948.06.patch
>
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have compactor
write directly to base_x_cZ or delta_x_y_cZ.  
> For query based compaction: can we control location of temp table dir?  We support external
temp tables so this may work but we'd need to have non-acid insert create files with {{bucket_xxxxx}}
names.
>  
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how retries of
tasks will work.  Just like we currently generate an MR job to compact, we should be able
to generate a Tez job.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message