hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <>
Subject [jira] [Commented] (HIVE-10458) Enable parallel order by for spark [Spark Branch]
Date Tue, 19 May 2015 11:44:00 GMT


Rui Li commented on HIVE-10458:

Hi [~xuefuz], we won't do double sample for approach a1. Because the {{TotalOrderPartitioner}}
is MR-specific.
One interesting thing I found is the qtest {{parallel_orderby.q}}. As I mentioned above, when
sorted data is stored in multiple files, we have to read these files in a proper order to
maintain the global sort. Seems when retrieving the results in FetchOperator, we rely on InputFormat::getSplits
which is related to the underlying FileSystem and doesn't guarantee an order. So if I run
{{parallel_orderby}} with local-cluster mode (TestSparkCliDriver), the FS used is LocalFileSystem
and it doesn't produce a correct result (in fact we do produce the correct results but we
don't read it in a proper way). However if I run {{parallel_orderby}} with yarn mode (TestMiniSparkOnYarnCliDriver),
the FS used is DistributedFileSystem and the result is correct. I also tried sorting the splits
in FetchOperator and then both modes work fine.
Maybe we should verify and fix this in a separate JIRA. What do you think?

> Enable parallel order by for spark [Spark Branch]
> -------------------------------------------------
>                 Key: HIVE-10458
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Rui Li
>            Assignee: Rui Li
>         Attachments: HIVE-10458.1-spark.patch, HIVE-10458.2-spark.patch, HIVE-10458.3-spark.patch
> We don't have to force reducer# to 1 as spark supports parallel sorting.

This message was sent by Atlassian JIRA

View raw message