spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: our spark gotchas report while creating batch pipeline
Date Sun, 18 Oct 2015 16:07:56 GMT
Interesting reading material.

bq. transformations that loose partitioner

lose partitioner

bq. Spark looses the partitioner

loses the partitioner

bq. Tunning number of partitions

Should be tuning.

bq. or increase shuffle fraction
bq. ShuffleMemoryManager: Thread 61 ...

Hopefully SPARK-10000 would alleviate the above situation.

Cheers

On Sun, Oct 18, 2015 at 8:51 AM, igor.berman <igor.berman@gmail.com> wrote:

> might be somebody will find it useful
> goo.gl/0yfvBd
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/our-spark-gotchas-report-while-creating-batch-pipeline-tp25112.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message