avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yin Huai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1208) Improve Trevni's performance on row-oriented data access
Date Fri, 15 Mar 2013 14:32:14 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13603413#comment-13603413
] 

Yin Huai commented on AVRO-1208:
--------------------------------

Here are some comparisons between RCFile and Trevni. I mainly focus on the data reading.

Based on current implementations, RCFile and Trevni have three major differences.
# The row group size in RCFile is configurable. Trevni uses a single row group for a Trevni
file (so applications need to horizontally partition the table to multiple Trevni files, and
the size of a Trevni file needs to be less than the HDFS block size. Am i right?).
# When reading needed columns of a row group, RCFile loads these columns at once. So, applications
need to wait the I/O (on reading all needed columns in a row group) before accessing any row
in this row group. In a Trevni file, a column is stored by many small blocks which are compression
units. When Trevni needs to read data from disks, applications only wait for Trevni to read
a few blocks before accessing a row.
# When reading needed columns of a row group, RCFile loads these columns in a column by column
fashion. For Trevni, applications need to decide how to read needed columns. They can read
data in a row by row fashion or in a column by column fashion.

For a given table, applications need to set a suitable row group size for RCFile. A small
row group size will cause a small size of a column in a row group (a column in a row group
is stored contiguously). Many seeks will degrade the performance of data reading (this is
described in Trevni specification). Also, a small row group size can cause a read buffer contain
data from unneeded columns and cause the OS readahead less effective (cannot asynchronous
fetch data from needed columns).

To overcome the low I/O efficiency of RCFile, a large row group size can be used. However,
RCFile needs to read all needed columns of a row group at once. In this way, CPU and I/O may
not be effectively overlapped (less benefit from OS asynchronous readahead). Suppose that
applications explicitly stores a table to multiple RCFile files and every file has a single
row group. When applications process data in a file, it will be blocked until all needed data
is loaded from disks. In this example, we will first wait on I/O and then wait on CPU.

For the third difference, a large row group size in RCFile imply a higher I/O performance
since all needed columns in a row group are read in a column by column fashion. But for Trevni,
since applications usually read data in needed columns in a row by row fashion (seems AvroColumnReader
reads data in a row by row fashion, and Hive and Pig integration of Trevni relies on this
reader), the throughput of reading data stored in Trevni can be significantly degraded (cased
by unnecessary disk seeks). 
                
> Improve Trevni's performance on row-oriented data access
> --------------------------------------------------------
>
>                 Key: AVRO-1208
>                 URL: https://issues.apache.org/jira/browse/AVRO-1208
>             Project: Avro
>          Issue Type: Improvement
>    Affects Versions: 1.7.3
>            Reporter: Yin Huai
>            Assignee: Yin Huai
>         Attachments: AVRO-1208.1.patch, AVRO-1208.2.patch
>
>
> Trevni uses an 64KB internal buffer to store values of a column. When accessing a column,
it reads 64KB (if we do not consider compression and checksum) data from the storage layer.
However, when the table is accessed in a row-oriented fashion (a entire row needs to be handed
over to the upper layer), in the worst case (a full table scan and values of this table are
all the same size), every 64KB data read can cause a seek.
> This jira is used to discuss if we should consider the data access pattern mentioned
above and if so, how to improve the performance of Trevni. Row-oriented data processing engines,
e.g. Hive, can benefit from this work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message