hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-20604) Minor compaction disables ORC column stats
Date Wed, 26 Sep 2018 18:33:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eugene Koifman updated HIVE-20604:
----------------------------------
    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

committed to master
thanks Prasanth for the review

> Minor compaction disables ORC column stats
> ------------------------------------------
>
>                 Key: HIVE-20604
>                 URL: https://issues.apache.org/jira/browse/HIVE-20604
>             Project: Hive
>          Issue Type: Improvement
>          Components: Transactions
>    Affects Versions: 1.0.0
>            Reporter: Eugene Koifman
>            Assignee: Eugene Koifman
>            Priority: Major
>             Fix For: 4.0.0
>
>         Attachments: HIVE-20604.01.patch
>
>
> {noformat}
>   @Override
>   public org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter
>         getRawRecordWriter(Path path, Options options) throws IOException {
>     final Path filename = AcidUtils.createFilename(path, options);
>     final OrcFile.WriterOptions opts =
>         OrcFile.writerOptions(options.getTableProperties(), options.getConfiguration());
>     if (!options.isWritingBase()) {
>       opts.bufferSize(OrcRecordUpdater.DELTA_BUFFER_SIZE)
>           .stripeSize(OrcRecordUpdater.DELTA_STRIPE_SIZE)
>           .blockPadding(false)
>           .compress(CompressionKind.NONE)
>           .rowIndexStride(0)
>       ;
>     }
> {noformat}
> {{rowIndexStride(0)}} makes {{StripeStatistics.getColumnStatistics()}} return objects
but with meaningless values, like min/max for {{IntegerColumnStatistics}} set to MIN_LONG/MAX_LONG.
> This interferes with ability to infer min ROW_ID for a split but also creates inefficient
files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message