spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-16015) Datasource register for shutdown?
Date Tue, 21 Jun 2016 12:39:57 GMT

    [ https://issues.apache.org/jira/browse/SPARK-16015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341687#comment-15341687
] 

Sean Owen commented on SPARK-16015:
-----------------------------------

On the executor? I don't think there's a lifecycle there to manage, because in theory it's
executing tasks that start and finish and don't have persistent state to clean up. Usually
you wouldn't hold open resources outside of a task.

I think you're doing the right thing. What's the issue?  Just some things log errors during
shutdown because things are shutting down? that may be to be expected. 

> Datasource register for shutdown?
> ---------------------------------
>
>                 Key: SPARK-16015
>                 URL: https://issues.apache.org/jira/browse/SPARK-16015
>             Project: Spark
>          Issue Type: Question
>          Components: Spark Core
>            Reporter: Michael Nitschinger
>
> Hi folks,
> I'm working on the couchbase datasource, but there is one thing that is bothering me
a little: Is there a way to get a notification from spark that shutdown is in progress? Ideally
I'd like to be able to clean up my resources before all the rest in spark shuts down.
> Right now I register a JVM shutdown hook to at least clean up the sockets but I've run
into issues like this (they go away if I manually call "stop" in my code before stopping the
spark context, but ideally I can do this transparently).
> {code}
> 2016-04-28 06:13:04 WARN  CouchbaseCore:166 - Exception while Handling Response Events
null
> java.lang.InterruptedException
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
>     at com.couchbase.client.deps.com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
>     at com.couchbase.client.deps.com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
>     at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>     at java.lang.Thread.run(Thread.java:745)
> 2016-04-28 06:13:04 WARN  CouchbaseCore:166 - Exception while Handling Request Events
RequestEvent{request=null}
> java.lang.InterruptedException
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
>     at com.couchbase.client.deps.com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
>     at com.couchbase.client.deps.com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
>     at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>     at java.lang.Thread.run(Thread.java:745)
> Exception: sbt.TrapExitSecurityException thrown from the UncaughtExceptionHandler in
thread "run-main-0"
> 2016-04-28 06:13:04 WARN  CouchbaseCore:166 - Exception while Handling Response Events
null
> java.lang.InterruptedException
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
>     at com.couchbase.client.deps.com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
>     at com.couchbase.client.deps.com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
>     at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>     at java.lang.Thread.run(Thread.java:745)
> 2016-04-28 06:13:04 WARN  CouchbaseCore:166 - Exception while Handling Request Events
RequestEvent{request=null}
> java.lang.InterruptedException
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
>     at com.couchbase.client.deps.com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
>     at com.couchbase.client.deps.com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
>     at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>     at java.lang.Thread.run(Thread.java:745)
> {code}
> Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message