hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-19418) add background stats updater similar to compactor
Date Wed, 06 Jun 2018 16:20:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503531#comment-16503531
] 

Hive QA commented on HIVE-19418:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 31s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 20s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 47s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m  8s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 27s{color} | {color:blue}
ql in master has 2280 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 32s{color} | {color:blue}
standalone-metastore in master has 214 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 45s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  8s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 16s{color} | {color:red}
hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 26s{color} | {color:red}
ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 47s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 47s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  9s{color} | {color:red}
itests/hcatalog-unit: The patch generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27)
{color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 33s{color} | {color:red}
ql: The patch generated 32 new + 120 unchanged - 0 fixed = 152 total (was 120) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 25s{color} | {color:red}
standalone-metastore: The patch generated 16 new + 1470 unchanged - 0 fixed = 1486 total (was
1470) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 34s{color} | {color:red}
ql generated 4 new + 2280 unchanged - 0 fixed = 2284 total (was 2280) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  2m 51s{color} | {color:red}
standalone-metastore generated 2 new + 213 unchanged - 1 fixed = 215 total (was 214) {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 50s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 11s{color} | {color:red}
The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m  7s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Dead store to writeIds in org.apache.hadoop.hive.ql.stats.StatsUpdaterThread.processOneTable(MetaStoreUtils$FullTableName)
 At StatsUpdaterThread.java:org.apache.hadoop.hive.ql.stats.StatsUpdaterThread.processOneTable(MetaStoreUtils$FullTableName)
 At StatsUpdaterThread.java:[line 221] |
|  |  Synchronization performed on java.util.concurrent.atomic.AtomicInteger in org.apache.hadoop.hive.ql.stats.StatsUpdaterThread.markAnalyzeDone(StatsUpdaterThread$AnalyzeWork)
 At StatsUpdaterThread.java:org.apache.hadoop.hive.ql.stats.StatsUpdaterThread.markAnalyzeDone(StatsUpdaterThread$AnalyzeWork)
 At StatsUpdaterThread.java:[line 501] |
|  |  Synchronization performed on java.util.concurrent.atomic.AtomicInteger in org.apache.hadoop.hive.ql.stats.StatsUpdaterThread.waitForQueuedCommands()
 At StatsUpdaterThread.java:org.apache.hadoop.hive.ql.stats.StatsUpdaterThread.waitForQueuedCommands()
 At StatsUpdaterThread.java:[line 642] |
|  |  org.apache.hadoop.hive.ql.stats.StatsUpdaterThread.buildPartColStr(Table) concatenates
strings using + in a loop  At StatsUpdaterThread.java:in a loop  At StatsUpdaterThread.java:[line
402] |
| FindBugs | module:standalone-metastore |
|  |  org.apache.hadoop.hive.metastore.ObjectStore$15.run(List) concatenates strings using
+ in a loop  At ObjectStore.java:in a loop  At ObjectStore.java:[line 8408] |
|  |  org.apache.hadoop.hive.metastore.ObjectStore$4.getJdoResult(ObjectStore$GetHelper) concatenates
strings using + in a loop  At ObjectStore.java:in a loop  At ObjectStore.java:[line 1628]
|
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-11562/dev-support/hive-personality.sh
|
| git revision | master / 0992d82 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-11562/yetus/patch-mvninstall-itests_hcatalog-unit.txt
|
| mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-11562/yetus/patch-mvninstall-ql.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-11562/yetus/diff-checkstyle-itests_hcatalog-unit.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-11562/yetus/diff-checkstyle-ql.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-11562/yetus/diff-checkstyle-standalone-metastore.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-11562/yetus/new-findbugs-ql.html
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-11562/yetus/new-findbugs-standalone-metastore.html
|
| asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-11562/yetus/patch-asflicense-problems.txt
|
| modules | C: itests/hcatalog-unit ql standalone-metastore U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-11562/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> add background stats updater similar to compactor
> -------------------------------------------------
>
>                 Key: HIVE-19418
>                 URL: https://issues.apache.org/jira/browse/HIVE-19418
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>            Priority: Major
>         Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, HIVE-19418.03.patch, HIVE-19418.04.patch,
HIVE-19418.05.patch, HIVE-19418.06.patch, HIVE-19418.06.patch, HIVE-19418.07.patch, HIVE-19418.07.patch,
HIVE-19418.patch
>
>
> There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables to make
them usable in a transaction without breaking ACID (for metadata-only optimization). However,
stats for ACID tables can still become unusable if e.g. two parallel inserts run - neither
sees the data written by the other, so after both finish, the snapshots on either set of stats
won't match the current snapshot and the stats will be unusable.
> Additionally, for ACID and non-ACID tables alike, a lot of the stats, with some exceptions
like numRows, cannot be aggregated (i.e. you cannot combine ndvs from two inserts), and for
ACID even less can be aggregated (you cannot derive min/max if some rows are deleted but you
don't scan the rest of the dataset).
> Therefore we will add background logic to metastore (similar to, and partially inside,
the ACID compactor) to update stats.
> It will have 3 modes of operation.
> 1) Off.
> 2) Update only the stats that exist but are out of date (generating stats can be expensive,
so if the user is only analyzing a subset of tables it should be able to only update that
subset). We can simply look at existing stats and only analyze for the relevant partitions
and columns.
> 3) On: 2 + create stats for all tables and columns missing stats.
> There will also be a table parameter to skip stats update. 
> In phase 1, the process will operate outside of compactor, and run analyze command on
the table. The analyze command will automatically save the stats with ACID snapshot information
if needed, based on HIVE-19416, so we don't need to do any special state management and this
will work for all table types. However it's also more expensive.
> In phase 2, we can explore adding stats collection during MM compaction that uses a temp
table. If we don't have open writers during major compaction (so we overwrite all of the data),
the temp table stats can simply be copied over to the main table with correct snapshot information,
saving us a table scan.
> In phase 3, we can add custom stats collection logic to full ACID compactor that is not
query based, the same way as we'd do for (2). Alternatively we can wait for ACID compactor
to become query based and just reuse (2).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message