hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
Date Tue, 16 Jul 2019 22:04:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886501#comment-16886501
] 

Hive QA commented on HIVE-20683:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 51s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m  6s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 35s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 57s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  6s{color} | {color:blue}
ql in master has 2249 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 31s{color} | {color:blue}
druid-handler in master has 3 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 20s{color} | {color:blue}
itests/qtest-druid in master has 7 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 22s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 27s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 43s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 41s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 41s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 40s{color} | {color:red}
ql: The patch generated 6 new + 35 unchanged - 0 fixed = 41 total (was 35) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 11s{color} | {color:red}
druid-handler: The patch generated 13 new + 0 unchanged - 0 fixed = 13 total (was 0) {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  2m  0s{color} | {color:red}
root: The patch generated 19 new + 35 unchanged - 0 fixed = 54 total (was 35) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  2s{color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 40s{color} | {color:red}
druid-handler generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 17s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 13s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 38s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:druid-handler |
|  |  Null passed for non-null parameter of new org.apache.druid.segment.virtual.ExpressionVirtualColumn(String,
String, ValueType, ExprMacroTable) in org.apache.hadoop.hive.druid.DruidStorageHandlerUtils.extractColName(ExprNodeDesc,
List)  At DruidStorageHandlerUtils.java:of new org.apache.druid.segment.virtual.ExpressionVirtualColumn(String,
String, ValueType, ExprMacroTable) in org.apache.hadoop.hive.druid.DruidStorageHandlerUtils.extractColName(ExprNodeDesc,
List)  At DruidStorageHandlerUtils.java:[line 1165] |
|  |  Switch statement found in org.apache.hadoop.hive.druid.DruidStorageHandlerUtils.addDynamicFilters(Query,
ExprNodeGenericFuncDesc, Configuration, boolean) where default case is missing  At DruidStorageHandlerUtils.java:ExprNodeGenericFuncDesc,
Configuration, boolean) where default case is missing  At DruidStorageHandlerUtils.java:[lines
974-1003] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  xml  compile  findbugs  checkstyle  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18051/dev-support/hive-personality.sh
|
| git revision | master / 65cdce1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18051/yetus/diff-checkstyle-ql.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18051/yetus/diff-checkstyle-druid-handler.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18051/yetus/diff-checkstyle-root.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-18051/yetus/new-findbugs-druid-handler.html
|
| modules | C: ql druid-handler . itests/qtest-druid U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18051/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Add the Ability to push Dynamic Between and Bloom filters to Druid
> ------------------------------------------------------------------
>
>                 Key: HIVE-20683
>                 URL: https://issues.apache.org/jira/browse/HIVE-20683
>             Project: Hive
>          Issue Type: New Feature
>          Components: Druid integration
>            Reporter: Nishant Bangarwa
>            Assignee: Nishant Bangarwa
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-20683.1.patch, HIVE-20683.patch
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM filter for
filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via https://github.com/apache/incubator-druid/pull/6222)
> Implementation details - 
> # Hive generates and passes the filters as part of 'filterExpr' in TableScan. 
> # DruidQueryBasedRecordReader gets this filter passed as part of the conf. 
> # During execution phase, before sending the query to druid in DruidQueryBasedRecordReader
we will deserialize this filter, translate it into a DruidDimFilter and add it to existing
DruidQuery.  Tez executor already ensures that when we start reading results from the record
reader, all the dynamic values are initialized. 
> # Explaining a druid query also prints the query sent to druid as {{druid.json.query}}.
We also need to make sure to update the druid query with the filters. During explain we do
not have the actual values for the dynamic values, so instead of values we will print the
dynamic expression itself as part of druid query. 
> Note:- This work needs druid to be updated to version 0.13.0



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message