spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "qihuagao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-21448) Hi dear guys, I have a question about aggregateByKey of pairrrd.
Date Wed, 19 Jul 2017 12:50:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

qihuagao updated SPARK-21448:
-----------------------------
    Description: 
java pair rdd has aggregateByKey, which can avoid full shuffle, so have impressive performance.
which has parameters, 
The aggregateByKey function requires 3 parameters:
# An intitial ‘zero’ value that will not effect the total values to be collected
# A combining function accepting two paremeters. The second paramter is merged into the first
parameter. This function combines/merges values within a partition.
# A merging function function accepting two parameters. In this case the parameters are merged
into one. This step merges values across partitions.

While Dataframe, I noticed groupByKey, which could do save function as aggregateByKey, but
without merge functions, so I assumed it should trigger shuffle operation. Is this true? if
true should we have a funtion like the performance like  aggregateByKey for dataframe?

Thanks.

  was:
java pair rrd has aggregateByKey, which can avoid full shuffle, so have impressive performance.
which has parameters, 
The aggregateByKey function requires 3 parameters:
# An intitial ‘zero’ value that will not effect the total values to be collected
# A combining function accepting two paremeters. The second paramter is merged into the first
parameter. This function combines/merges values within a partition.
# A merging function function accepting two parameters. In this case the parameters are merged
into one. This step merges values across partitions.

While Dataframe, I noticed groupByKey, which could do save function as aggregateByKey, but
without merge functions, so I assumed it should trigger shuffle operation. Is this true? if
true should we have a funtion like the performance like  aggregateByKey for dataframe?

Thanks.


> Hi dear guys,  I have a question about aggregateByKey of pairrrd.
> -----------------------------------------------------------------
>
>                 Key: SPARK-21448
>                 URL: https://issues.apache.org/jira/browse/SPARK-21448
>             Project: Spark
>          Issue Type: Question
>          Components: Java API
>    Affects Versions: 2.0.0
>         Environment: Spark 2.0
>            Reporter: qihuagao
>
> java pair rdd has aggregateByKey, which can avoid full shuffle, so have impressive performance.
which has parameters, 
> The aggregateByKey function requires 3 parameters:
> # An intitial ‘zero’ value that will not effect the total values to be collected
> # A combining function accepting two paremeters. The second paramter is merged into the
first parameter. This function combines/merges values within a partition.
> # A merging function function accepting two parameters. In this case the parameters are
merged into one. This step merges values across partitions.
> While Dataframe, I noticed groupByKey, which could do save function as aggregateByKey,
but without merge functions, so I assumed it should trigger shuffle operation. Is this true?
if true should we have a funtion like the performance like  aggregateByKey for dataframe?
> Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message