hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-22819) Refactor Hive::listFilesCreatedByQuery to make it faster for object stores
Date Tue, 04 Feb 2020 12:52:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-22819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029823#comment-17029823
] 

Hive QA commented on HIVE-22819:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  1s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 52s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 57s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 23s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 55s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 46s{color} | {color:blue}
ql in master has 1533 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 28s{color} | {color:blue}
hcatalog/streaming in master has 11 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 28s{color} | {color:blue}
streaming in master has 2 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 44s{color} | {color:blue}
itests/hive-unit in master has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 47s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 29s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 54s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 31s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 31s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m  6s{color} | {color:red}
ql: The patch generated 47 new + 2707 unchanged - 24 fixed = 2754 total (was 2731) {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 20s{color} | {color:red}
itests/hive-unit: The patch generated 22 new + 177 unchanged - 21 fixed = 199 total (was 198)
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m  2s{color} | {color:red}
ql generated 1 new + 1532 unchanged - 1 fixed = 1533 total (was 1533) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 51s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 14s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 39s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Dead store to filename in org.apache.hadoop.hive.ql.io.AcidUtils.parseBucketId(Path)
 At AcidUtils.java:org.apache.hadoop.hive.ql.io.AcidUtils.parseBucketId(Path)  At AcidUtils.java:[line
434] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20456/dev-support/hive-personality.sh
|
| git revision | master / 06c5923 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20456/yetus/diff-checkstyle-ql.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20456/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-20456/yetus/new-findbugs-ql.html
|
| modules | C: ql hcatalog/streaming streaming itests/hive-unit U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20456/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Refactor Hive::listFilesCreatedByQuery to make it faster for object stores
> --------------------------------------------------------------------------
>
>                 Key: HIVE-22819
>                 URL: https://issues.apache.org/jira/browse/HIVE-22819
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Marton Bod
>            Assignee: Marton Bod
>            Priority: Major
>         Attachments: HIVE-22819.1.patch, HIVE-22819.2.patch
>
>
> {color:#0000ff}Hive::listFilesCreatedByQuery{color} does an exists(), an isDir() and
then a listing call. This can be expensive in object stores. We should instead directly list
the files in the directory (we'd have to handle an exception if the directory does not exists,
but issuing a single call to the object store would most likely still end up being more performant). 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message