spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gatorsmile <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-12922][SparkR][WIP] Implement gapply() ...
Date Mon, 02 May 2016 23:12:23 GMT
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12836#discussion_r61820422
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
    @@ -113,6 +116,46 @@ class KeyValueGroupedDataset[K, V] private[sql](
       }
     
       /**
    +   * Applies the given R function to each group of data.  For each unique group, the
function will
    +   * be passed the group key and an iterator that contains all of the elements in the
group. The
    +   * function can return an iterator containing elements of an arbitrary type which will
be returned
    +   * as a new [[Dataset]].
    +   *
    +   * This function does not support partial aggregation, and as a result requires shuffling
all
    +   * the data in the [[Dataset]]. If an application intends to perform an aggregation
over each
    +   * key, it is best to use the reduce function or an
    +   * [[org.apache.spark.sql.expressions#Aggregator Aggregator]].
    +   *
    +   * Internally, the implementation will spill to disk if any given group is too large
to fit into
    +   * memory.  However, users must take care to avoid materializing the whole iterator
for a group
    +   * (for example, by calling `toList`) unless they are sure that this is possible given
the memory
    +   * constraints of their cluster.
    +   *
    +   * @since 2.0.0
    +   */
    +  def flatMapRGroups(
    +    f: Array[Byte],
    +    packageNames: Array[Byte],
    +    broadcastVars: Array[Object],
    +    outputSchema: StructType): DataFrame = {
    --- End diff --
    
    four spaces


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message