spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Peloton (JIRA)" <>
Subject [jira] [Created] (SPARK-26024) Dataset API: repartitionByRange(...) has inconsistent behaviour
Date Mon, 12 Nov 2018 21:49:00 GMT
Julien Peloton created SPARK-26024:

             Summary: Dataset API: repartitionByRange(...) has inconsistent behaviour
                 Key: SPARK-26024
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.2
         Environment: Spark version 2.3.2
            Reporter: Julien Peloton


I recently played with the {{repartitionByRange}} method for DataFrame introduced in SPARK-22614.
For DataFrames larger than the one tested in the code (which has only 10 elements), the code
sends back random results.

As a test for showing the inconsistent behaviour, I start as the unit code used to test {{repartitionByRange}}
but I increase the size of the initial array to 1000, repartition using 3 partitions, and
count the number of element per-partitions:

// Shuffle numbers from 0 to 1000, and make a DataFrame
val df = Random.shuffle("val")

// Repartition it using 3 partitions
// Sum up number of elements in each partition, and collect it.
// And do it several times
for (i <- 0 to 9) {
  var counts = df.repartitionByRange(3, col("val"))
    .mapPartitions{part => Iterator(part.size)}
// -> the number of elements in each partition varies...
I do not know whether it is expected (I will dig further in the code), but it sounds like
a bug.
 Or I just misinterpret what {{repartitionByRange}} is for?
 Any ideas?


This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message