hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (JIRA)" <>
Subject [jira] [Commented] (HIVE-3874) Create a new Optimized Row Columnar file format for Hive
Date Thu, 21 Feb 2013 18:00:20 GMT


Phabricator commented on HIVE-3874:

omalley has commented on the revision "HIVE-3874 [jira] Create a new Optimized Row Columnar
file format for Hive".

  | Do you want to have a simple test for HIVE-4015 as part of this patch ?

  Since the test will fail until I change the grammer, I think it would be better to wait
until they will pass.

  | 1. Can you add more comments - specially, in the class/interface definitions Writer/TreeWriter/StreamFactory
to name a few.


  | 2. Can column statistics be made optional ? (can be a follow-up)

  They are very cheap in practice, but it wouldn't be hard to disable them.

  | 3. This has a lot of new code - I mean, is it possible to use some of the constructs which
are already there - for eg. RedBlackTrees, RLE etc. Can you use some existing implementations
instead of writing these from scratch ?

  I'm a big fan of not writing new code when I can just use someone else's. That said, it
is always a trade off when evaluating when to re-use versus write new that involves comparing
the requirements to what the other code provides.

  I'm not aware of any open source java red-black trees that work on primitives without allocating
multiple objects per an entry. Do you have a suggestion?

  The RLE is very specific to ORC format and it didn't seem like any of the implementations
available were a good match. I'm also considering how to do a better delta and small integer
encoding, but I'll do that in a follow up jira.

  | Right now, the RLE is fixed. Should it be pluggable ? I mean - we can have a different
scheme to store deltas.

  I think that pluggable will create compatibility problems, since you won't be able to read
an ORC file that was written by a different plugin.

  My preferred direction is to use the ColumnEncoding to allow the Writer to pick a different
encoding based on the observed data. For example, by looking at the first 100,000 values the
writer should be able to decide if a dictionary or direct encoding is better. We could use
the same mechanism to add additional encodings.

  ql/src/java/org/apache/hadoop/hive/ql/orc/ There is an requirement
that the codec's compress method will return false rather than take more space than the input.
Given that, if the compressed is empty, we can't need the overflow.
  ql/src/java/org/apache/hadoop/hive/ql/orc/ I've removed it.
  ql/src/java/org/apache/hadoop/hive/ql/orc/ fixed.
  ql/src/java/org/apache/hadoop/hive/ql/orc/ I've added the size of
the dictionary to the estimate of the memory size, which should be better.
  ql/src/java/org/apache/hadoop/hive/ql/orc/ i managed to move the directory
in the wrong place. fixed.


To: JIRA, omalley
Cc: kevinwilfong, njain

> Create a new Optimized Row Columnar file format for Hive
> --------------------------------------------------------
>                 Key: HIVE-3874
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: Serializers/Deserializers
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: hive.3874.2.patch, HIVE-3874.D8529.1.patch, HIVE-3874.D8529.2.patch,
HIVE-3874.D8529.3.patch, OrcFileIntro.pptx, orc.tgz
> There are several limitations of the current RC File format that I'd like to address
by creating a new format:
> * each column value is stored as a binary blob, which means:
> ** the entire column value must be read, decompressed, and deserialized
> ** the file format can't use smarter type-specific compression
> ** push down filters can't be evaluated
> * the start of each row group needs to be found by scanning
> * user metadata can only be added to the file when the file is created
> * the file doesn't store the number of rows per a file or row group
> * there is no mechanism for seeking to a particular row number, which is required for
external indexes.
> * there is no mechanism for storing light weight indexes within the file to enable push-down
filters to skip entire row groups.
> * the type of the rows aren't stored in the file

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message