hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-3874) Create a new Optimized Row Columnar file format for Hive
Date Thu, 21 Feb 2013 17:44:17 GMT

     [ https://issues.apache.org/jira/browse/HIVE-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Phabricator updated HIVE-3874:
------------------------------

    Attachment: HIVE-3874.D8529.3.patch

omalley updated the revision "HIVE-3874 [jira] Create a new Optimized Row Columnar file format
for Hive".

    - fix unit tests

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D8529

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D8529?vs=28101&id=28305#toc

AFFECTED FILES
  build.properties
  build.xml
  ivy/libraries.properties
  ql/build.xml
  ql/ivy.xml
  ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldWriter.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/BooleanColumnStatistics.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatistics.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/CompressionCodec.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/CompressionKind.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/DoubleColumnStatistics.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicByteArray.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicIntArray.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileDump.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/IntegerColumnStatistics.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcStruct.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcUnion.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OutStream.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/PositionProvider.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/PositionRecorder.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/PositionedOutputStream.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReader.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RedBlackTree.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteWriter.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReader.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriter.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/SerializationUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/StreamName.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringColumnStatistics.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringRedBlackTree.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/StripeInformation.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/package-info.java
  ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestDynamicArray.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestFileDump.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcStruct.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestSerializationUtils.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestStreamName.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestStringRedBlackTree.java
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestZlib.java
  ql/src/test/resources/orc-file-dump.out

To: JIRA, omalley
Cc: kevinwilfong, njain

                
> Create a new Optimized Row Columnar file format for Hive
> --------------------------------------------------------
>
>                 Key: HIVE-3874
>                 URL: https://issues.apache.org/jira/browse/HIVE-3874
>             Project: Hive
>          Issue Type: Improvement
>          Components: Serializers/Deserializers
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: hive.3874.2.patch, HIVE-3874.D8529.1.patch, HIVE-3874.D8529.2.patch,
HIVE-3874.D8529.3.patch, OrcFileIntro.pptx, orc.tgz
>
>
> There are several limitations of the current RC File format that I'd like to address
by creating a new format:
> * each column value is stored as a binary blob, which means:
> ** the entire column value must be read, decompressed, and deserialized
> ** the file format can't use smarter type-specific compression
> ** push down filters can't be evaluated
> * the start of each row group needs to be found by scanning
> * user metadata can only be added to the file when the file is created
> * the file doesn't store the number of rows per a file or row group
> * there is no mechanism for seeking to a particular row number, which is required for
external indexes.
> * there is no mechanism for storing light weight indexes within the file to enable push-down
filters to skip entire row groups.
> * the type of the rows aren't stored in the file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message