hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal Vijayaraghavan <gop...@apache.org>
Subject Re: reading ORC format on Spark-SQL
Date Wed, 10 Feb 2016 21:42:51 GMT

> The reason why I am asking this kind of question is reading csv file on
>Spark is linearly increasing as the data size increase a bit, but reading
>ORC format on Spark-SQL is still same as the data size increses in
><figure 2>.
...
> This cause is from (just property of reading ORC format) or (creating
>the table for input and loading the input in the table) or both?

ORC readers are more efficient than reading text, but ORC readers cannot
split beyond a 64Mb chunk, while text readers can split down to 1 line per
task.

So, it's possible the CSV readers are producing many many more divisions
and running the query using the full cluster always - splitting
indiscriminately is not always faster as each task has some fixed overhead
unrelated to the data size (like plan deserialization in Kryo).

For ORC - 59 tasks can run in the same time as 193 tasks, as long as
there's capacity to run 193 in a single pass (like 200 executors).

Until you run out of capacity, a distributed system *has* to show
sub-linear scaling - and will show flat scaling upto a particular point
because of Amdahl's law.

Cheers,
Gopal



Mime
View raw message