spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-16395) Fail if too many CheckpointWriteHandlers are queued up in the fixed thread pool
Date Tue, 19 Jul 2016 11:12:20 GMT

     [ https://issues.apache.org/jira/browse/SPARK-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen updated SPARK-16395:
------------------------------
    Assignee: Sean Owen

> Fail if too many CheckpointWriteHandlers are queued up in the fixed thread pool
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-16395
>                 URL: https://issues.apache.org/jira/browse/SPARK-16395
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Yan Chen
>            Assignee: Sean Owen
>             Fix For: 2.1.0
>
>
> When the batch interval is small and time spent writing to HDFS for checkpointing is
always longer than the batch interval, the CheckpointWriteHandlers could be queued up in the
fixed thread pool. 
> Memory issue could happen in this situation.
> So if we can log a warning if this situation happens, it will help a lot of people when
they encounter the same problem.
> The idea is if the size of the queue exceeds some number, we log a warning saying it's
queued up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message