hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20517) Creation of staging directory and Move operation is taking time in S3
Date Mon, 17 Sep 2018 06:18:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617125#comment-16617125
] 

Hive QA commented on HIVE-20517:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 35s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 49s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 22s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 27s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 32s{color} | {color:blue}
common in master has 65 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 40s{color} | {color:blue}
itests/hive-unit in master has 2 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  1s{color} | {color:blue}
ql in master has 2326 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 14s{color} | {color:red}
metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 56s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  9s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 39s{color} | {color:red}
hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 29s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 29s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 24s{color} | {color:red}
itests/hive-unit: The patch generated 55 new + 695 unchanged - 1 fixed = 750 total (was 696)
{color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 44s{color} | {color:red}
ql: The patch generated 4 new + 420 unchanged - 3 fixed = 424 total (was 423) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 15s{color} | {color:red}
metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 57s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 14s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 48s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-13850/dev-support/hive-personality.sh
|
| git revision | master / a37827e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-13850/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-13850/yetus/patch-mvninstall-itests_hive-unit.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-13850/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-13850/yetus/diff-checkstyle-ql.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-13850/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: common itests/hive-unit ql standalone-metastore/metastore-server U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-13850/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Creation of staging directory and Move operation is taking time in S3
> ---------------------------------------------------------------------
>
>                 Key: HIVE-20517
>                 URL: https://issues.apache.org/jira/browse/HIVE-20517
>             Project: Hive
>          Issue Type: Sub-task
>          Components: repl
>    Affects Versions: 4.0.0
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>         Attachments: HIVE-20517.01.patch, HIVE-20517.02.patch, HIVE-20517.03.patch, HIVE-20517.04.patch,
HIVE-20517.05.patch
>
>
> Operations like insert and add partition creates a staging directory to generate the
files and then move the files created to actual location. In replication flow, the files are
first copied to the staging directory and then moved (rename) to the actual table location.
In case of S3, move is not an atomic operation. It internally does a copy and delete. So it
can not guarantee the consistency required. So it is better to copy the files directly to
the actual location. This will help in avoiding the staging directory creation (which takes
1-2 seconds in s3) and move (which takes time proportional to file size).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message