orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal Vijayaraghavan <gop...@apache.org>
Subject Re: ORC double encoding optimization proposal
Date Mon, 26 Mar 2018 06:47:22 GMT

>    2. Under seek or predicate pushdown scenario, there’s no need to load the entire
Yes, that is a valid scenario where the reader reads partial-streams & causes random IO.

The current double encoding is actually 2 streams today & will continue to use 2 streams
for the FLIP implementation.

The SPLIT implementation will go from the current 2 streams to 4 streams (i.e 1+1->1+3
streams) & the total data IO will drop by ~2x or so. More so if one of the streams can
be suppressed (like in my IoT data-set, where the sign-bit is always +ve for my electric meter

The trade-offs seem to be working out on regular HDDs with locality & for LLAP SSD caches
- if your use-cases are different, I'd like to hear more about it.

The only significant random IO delays expected seem to be entirely within the HDFS API network
hops (which offers 0% locality when data is erasure coded or for cloud-storage), which I hope
to fix in the Hadoop-3.x branch with a new API.


View raw message