orc-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Reading ORC Files from S3
Date Mon, 28 Sep 2015 20:44:41 GMT
A colleague of mine posted to this list a few months ago about some 
difficulties we were experiencing reading from ORC files stored on 
Amazon S3.  What we were finding was that a set of ORC files that we 
built performed well on HDFS, but showed extremely poor performance when 
stored on S3.  I've been continuing my colleague's work, and have tried 
various and sundry fixes and tweaks to try to get the performance to 
improve, but so far to no avail.  I was hoping perhaps someone on the 
list here might be able to shed some light as to why we're having these 
problems and/or have some suggestions on how we might be able to work 
around them.

A bit more details about our issues:

We have 2 datasets that we've built which are stored as ORC files.  The 
first set is a series of records, sorted by record ID.  The second set 
is an inverted index into the first set, where each record contains a 
search key value followed by a record ID.  (The 2nd dataset is sorted by 
search key value.)  The first dataset contains ~4000 files, totaling 
500GB (i.e., ~120MB per file); the second also contains ~4000 files, but 
totaling nearly 2TB (~230MB per file).

What I'm finding is that queries against the first dataset (the records) 
complete in a fairly reasonable amount of time, but queries against the 
index dataset are taking a very long time.  This is completely contrary 
to what I would expect, as the index dataset should be better able to 
take advantage of the efficiencies built into the ORC data storage, and 
so should be able to be queried faster.  (I.e., theoretically ORC should 
be able to skip reading large portions of the index files by jumping 
directly to the index records that match the supplied search criteria. 
(Or at least jumping to a stripe close to them.))  But this is proving 
not to be the case.

All of the ORC files are generated using a custom map/reduce job with 
OrcNewOutputFormat (using Hive 0.13.1 jars) and are being queried via 
Hive queries (using Hive 1.1.0).  The files are initially written to 
HDFS, and then pushed to S3 (using distcp).  But my queries are all 
being done directly against the files stored on S3.  (I.e., a Hive 
external table with a LOCATION pointing to S3.)

I've tried various tweaks to the ORC file generation process - larger 
number of small files, smaller number of large files, stripe sizes 
varying from 64MB to 256MB, etc.  But nothing seems to make any 
difference.  Queries against the index dataset take a very long time no 
matter what I try - as in 4x-5x longer than querying the records dataset.

One other thing that I'm finding particularly strange here is that 
enabling predicate pushdown is seeming to have no effect here - and 
sometimes even makes things worse.  When I set 
"hive.optimize.index.filter=true" I can see that the predicate pushdown 
is taking effect via output in the Hadoop job logs.  But it doesn't seem 
like the predicate pushdown is able to make the query run any faster 
when the data is held on S3.

ORC isn't giving me much clue as to the cause for the delays either. 
When I look in the Hadoop job task logs, I see a message about the 
S3NativeFileSystem opening one of my ORC files ... and then 6-7 minutes 
pass before I see the next log message about Hive starting to process 
the records in the file.

One other thing I've noticed is that I don't seem to be the only one 
experiencing this issue.  Googling on this topic turned up a few other 
people with a similar problem, most notably the blog post at 
http://bitmusings.tumblr.com/post/56081787247/orc-files-in-hive where 
the author wound up finding the performance so bad that he switched from 
using S3 native storage format to using the S3 block storage format in 
order to work around these issues.

So .... anyone have any ideas as to what might be causing this issue 
and/or how to work around?  Is ORC simply unable to work efficiently 
against data stored on S3n?  (I.e., due to network round-trips taking 
too long.)  Any help anyone could offer would be greatly appreciated! 
This is proving to be a blocker issue for my project, and if I can't 
find a solution I'm likely going to wind up having to scrap the idea of 
using ORC to store the index.




View raw message