streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Blackmon (JIRA)" <>
Subject [jira] [Created] (STREAMS-183) streams-local closes persist-writers before datums in-flight are written
Date Sun, 28 Sep 2014 17:51:33 GMT
Steve Blackmon created STREAMS-183:

             Summary: streams-local closes persist-writers before datums in-flight are written
                 Key: STREAMS-183
             Project: Streams
          Issue Type: Bug
            Reporter: Steve Blackmon

I have observed this behavior when writing datums from a readCurrent provider through to ElasticsearchPersistWriter
and ElasticsearchPersistDeleter.  

The log looks like:

421777 [pool-2-thread-7] INFO  o.a.s.e.ElasticsearchPersistReader - PersistReader done
469271 [main] INFO - Attempting to shutdown tasks
476319 [pool-2-thread-3] ERROR o.a.s.e.ElasticsearchPersistWriter - There was an error sending
the batch: No node available
485275 [main] WARN - Task org.apache.streams.local.tasks.StreamsPersistWriterTask@6add646d
failed to terminate in allotted timeframe
495276 [pool-2-thread-3] WARN  o.a.s.e.ElasticsearchPersistWriter - Catchup was interrupted.
 Data may be lost
followed by additional No Node Available exceptions.

When the same stream is executed as a perpetual stream, this problem does not occur.

Possible solutions: 
a) localstreambuilder waits a while between detecting providers have completed and stopping
downstream components
b) localstreambuilder does not stop downstream components until all datums have been processed/written
c) writers refuse to close until all datums have been written

This message was sent by Atlassian JIRA

View raw message