hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Hanson (JIRA)" <>
Subject [jira] [Updated] (HIVE-6234) Implement fast vectorized InputFormat extension for text files
Date Fri, 31 Jan 2014 23:08:09 GMT


Eric Hanson updated HIVE-6234:

    Attachment: state-diagram.jpg

State diagram for finding line breaks. May be of use for future reference. Not done. Just
a working document.

> Implement fast vectorized InputFormat extension for text files
> --------------------------------------------------------------
>                 Key: HIVE-6234
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Eric Hanson
>            Assignee: Eric Hanson
>         Attachments: HIVE-6234.02.patch, HIVE-6234.03.patch, Vectorized Text InputFormat
design.docx, Vectorized Text InputFormat design.pdf, state-diagram.jpg
> Implement support for vectorized scan input of text files (plain text with configurable
record and field separators). This should work for CSV files, tab delimited files, etc. 
> The goal is to provide high-performance reading of these files using vectorized scans,
and also to do it as an extension of existing Hive. Then, if vectorized query is enabled,
existing tables based on text files will be able to benefit immediately without the need to
use a different input format. After upgrading to new Hive bits that support this, faster,
vectorized processing over existing text tables should just work, when vectorization is enabled.
> Another goal is to go beyond a simple layering of vectorized row batch iterator over
the top of the existing row iterator. It should be possible to, say, read a chunk of data
into a byte buffer (several thousand or even million rows), and then read data from it into
vectorized row batches directly. Object creations should be minimized to save allocation time
and GC overhead. If it is possible to save CPU for values like dates and numbers by caching
the translation from string to the final data type, that should ideally be implemented.

This message was sent by Atlassian JIRA

View raw message