spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Berman <igor.ber...@gmail.com>
Subject Re: our spark gotchas report while creating batch pipeline
Date Sun, 18 Oct 2015 20:29:24 GMT
thanks Ted :)


On 18 October 2015 at 19:07, Ted Yu <yuzhihong@gmail.com> wrote:

> Interesting reading material.
>
> bq. transformations that loose partitioner
>
> lose partitioner
>
> bq. Spark looses the partitioner
>
> loses the partitioner
>
> bq. Tunning number of partitions
>
> Should be tuning.
>
> bq. or increase shuffle fraction
> bq. ShuffleMemoryManager: Thread 61 ...
>
> Hopefully SPARK-10000 would alleviate the above situation.
>
> Cheers
>
> On Sun, Oct 18, 2015 at 8:51 AM, igor.berman <igor.berman@gmail.com>
> wrote:
>
>> might be somebody will find it useful
>> goo.gl/0yfvBd
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/our-spark-gotchas-report-while-creating-batch-pipeline-tp25112.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Mime
View raw message