spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Fregly (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-2475) Check whether #cores > #receivers in local mode
Date Thu, 28 Aug 2014 19:24:09 GMT

    [ https://issues.apache.org/jira/browse/SPARK-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114204#comment-14114204
] 

Chris Fregly commented on SPARK-2475:
-------------------------------------

another option for the examples, specifically, is to default the number of local threads similar
to to how the Kinesis example does it:  

https://github.com/apache/spark/blob/ae58aea2d1435b5bb011e68127e1bcddc2edf5b2/extras/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala#L104

i get the number of shards in the given Kinesis stream and add 1.  the goal was to make this
example work out of the box with little friction - even an error message can be discouraging.

for the other examples, we could just default to 2.  the advanced user can override if they
want.  though i don't think i support an override in my kinesis example.  whoops!  :)

> Check whether #cores > #receivers in local mode
> -----------------------------------------------
>
>                 Key: SPARK-2475
>                 URL: https://issues.apache.org/jira/browse/SPARK-2475
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Tathagata Das
>
> When the number of slots in local mode is not more than the number of receivers, then
the system should throw an error. Otherwise the system just keeps waiting for resources to
process the received data.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message