spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-22606) There may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-threaded access"
Date Mon, 11 Dec 2017 21:23:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-22606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shixiong Zhu updated SPARK-22606:
---------------------------------
    Component/s:     (was: Structured Streaming)
                 DStreams

> There may be two or more tasks in one executor will use the same kafka consumer at the
same time, then it will throw an exception: "KafkaConsumer is not safe for multi-threaded
access" 
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-22606
>                 URL: https://issues.apache.org/jira/browse/SPARK-22606
>             Project: Spark
>          Issue Type: Bug
>          Components: DStreams
>    Affects Versions: 2.2.0
>            Reporter: eaton
>            Priority: Minor
>
> If the value of param 'spark.streaming.concurrentJobs' is more than one, and the value
of param 'spark.executor.cores' is more than one, there may be two or more tasks in one executor
will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer
is not safe for multi-threaded access" 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message