nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alessandro D'Armiento (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (NIFI-6513) The property of PutHive3Streaming Processor named "Auto-Create Partitions" is useless
Date Sat, 03 Aug 2019 11:42:00 GMT

    [ https://issues.apache.org/jira/browse/NIFI-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899443#comment-16899443
] 

Alessandro D'Armiento edited comment on NIFI-6513 at 8/3/19 11:41 AM:
----------------------------------------------------------------------

Are we sure we actually need that boolean? 
In case you enable dynamic partitioning, you should be able to set the maximum number of partitions
in the hive configuration files you are able to pass to the `Hive Configuration Resources`
property. Furthermore, HiveConnection[1] isDynamic() method only checks for the existence
of a list of static partitions (as well as if the table itself is partitioned). 
If you instead filled the `Partition Values` you are telling Hive you are using static partitioning
and no partitions should be created anyhow. 

I suggest renaming `Partition Values` with `Static Partition Values` for correctness and remove
that boolean

cc [~mattyb149], which worked on the Hive Processor in NIFI-4963

[1] https://github.com/apache/hive/blob/d7475aa98f6a2fc813e2e1c0ad99f902cb28cc00/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java#L867


was (Author: axelsync):
Are we sure we actually need that boolean? 
In case you enable dynamic partitioning, you should be able to set the maximum number of partitions
in the hive configuration files you are able to pass to the `Hive Configuration Resources`
property. If you instead filled the `Partition Values` you are telling Hive you are using
static partitioning and no partitions should be created anyhow. 

I suggest renaming `Partition Values` with `Static Partition Values` for correctness and remove
that boolean

cc [~mattyb149], which worked on the Hive Processor in NIFI-4963

> The property of PutHive3Streaming Processor named "Auto-Create Partitions" is useless
> -------------------------------------------------------------------------------------
>
>                 Key: NIFI-6513
>                 URL: https://issues.apache.org/jira/browse/NIFI-6513
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.9.0
>         Environment: Hive3.1.0
>            Reporter: Lu Wang
>            Priority: Major
>
> _emphasized text_The PutHive3Streaming processor always creates the partition, regardless
of whether the  value of property named "Auto-Create Partitions" is set to true or false。



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message