spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Commented] (SPARK-21369) Don't use Scala classes in external shuffle service
Date Mon, 10 Jul 2017 20:48:00 GMT


Apache Spark commented on SPARK-21369:

User 'zsxwing' has created a pull request for this issue:

> Don't use Scala classes in external shuffle service
> ---------------------------------------------------
>                 Key: SPARK-21369
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle, YARN
>    Affects Versions: 2.2.0
>            Reporter: Shixiong Zhu
>            Assignee: Shixiong Zhu
> Right now the external shuffle service uses Scala Tuple2. However, the Scala library
won't be shaded into the yarn shuffle assembly jar. Then when the codes are called, it will
throw ClassNotFoundException.
> Right now it's safe by default because we disabled spark.reducer.maxReqSizeShuffleToMem.
However,  to allow using spark.reducer.maxReqSizeShuffleToMem, we should remove all usages
of Tuples.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message