orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ORC-74) The ORC writer should suppress the indexes when they are redundant
Date Fri, 01 Jul 2016 20:28:10 GMT
Owen O'Malley created ORC-74:
--------------------------------

             Summary: The ORC writer should suppress the indexes when they are redundant
                 Key: ORC-74
                 URL: https://issues.apache.org/jira/browse/ORC-74
             Project: Orc
          Issue Type: Improvement
            Reporter: Owen O'Malley


Currently we always include column indexes and statistics at three levels:
* file
* stripe
* row group (10k rows)

At each level, there isn't any value if the higher level index is exactly the same. Thus,
if there is a single stripe in a file, we shouldn't have stripe statistics. If there is a
single row group in the stripe, we shouldn't include the row group indexes.

This should improve use cases like Apache NiFi where each file often has less than 10k rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message