streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (STREAMS-81) Monitor thread pool never fully shuts down tasks in LocalStreamBuilder
Date Sat, 17 May 2014 04:59:15 GMT

    [ https://issues.apache.org/jira/browse/STREAMS-81?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000668#comment-14000668
] 

ASF GitHub Bot commented on STREAMS-81:
---------------------------------------

GitHub user smashew opened a pull request:

    https://github.com/apache/incubator-streams/pull/16

    Overhaul of LocalStreamsBuilder

    Commit Fixes:
    https://issues.apache.org/jira/browse/STREAMS-81
    https://issues.apache.org/jira/browse/STREAMS-80
    https://issues.apache.org/jira/browse/STREAMS-82
    https://issues.apache.org/jira/browse/STREAMS-83
    https://issues.apache.org/jira/browse/STREAMS-84
    https://issues.apache.org/jira/browse/STREAMS-85
    
    Very large overhaul, pardon the large commit. The only major addition to this is to allow
for the 'concept' of a known finite stream. There are many streams that are listened to and
will be opened for a long time, however, there are other streams, that have a known termination
point. 
    
    IE: Reading 1,000,000 records from an index in ElasticSearch to HDFS
    
    There is a flag, now in StreamsResultSet, that when set to true, changes the 'queue' from
being read as a Collection<E> of items to an actual shared queue. This has many benefits,
mainly, the queue is respected as a queue instead of being treated as a batch. In paradigms
where items can be read in a stream, via a queue, this is an excellent choice.
    
    There were many bugs that prevented the execution of this software from being run inside
of a currently running JVM as it would cause memory and thread leaks. This has been updated
and repaired in this version. 
    
    +++ tests +++
    The tests that were in LocalStreamsBuilder  have been restored and updated to be accurate
(many of the tests were not accurate). This helps to ensure backward compatibility with the
previous implementation. In addition, many (20) tests were added for edge cases and various
amounts of behavior that could disrupt the execution of a stream. (RuntimeExceptions, excessive
delays in processing, etc...) All tests have been written and are currently passing. 
    
    Thank you for your time!
    Smashew

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/smashew/incubator-streams master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-streams/pull/16.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #16
    
----
commit c21e2b8687738753ff51ff2a3e595a1e4b5a439c
Author: Matthew.Hager@gmail.com <matthew.hager@gmail.com>
Date:   2014-05-13T16:19:45Z

    Merge pull request #1 from apache/master
    
    f

commit 5c6f6560c2a9d90ad1406733f0dee80b1f418804
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-15T16:19:39Z

    STREAMS-81 - Fix for:  There is an issue shutting down the local streambuilder, this is
the fix.

commit 1c30164a2d58d141b15d7ea181405542eda766e2
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-15T16:32:21Z

    I noticed these threads running to, causing an issue.

commit ff6bbd579594a20893ae33a64b11c1cb7e085b38
Author: Matthew.Hager@gmail.com <matthew.hager@gmail.com>
Date:   2014-05-15T20:39:14Z

    Merge pull request #2 from apache/master
    
    merge

commit 49511e8022627e8ee1e2257372a354c753477868
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-16T01:01:59Z

    added some more tests, re

commit bf6e4627957b845d74b19e50306f6c3168d8a62f
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-16T15:39:38Z

    Added more tests... still working on getting them all to pass and adding some more delay
tests

commit 4e5e78d550559235e47c9fc7f3b4bb9816825e66
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-16T15:40:26Z

    Changes to url

commit 2683fc8617edbd86b1258e48d5cb873f29e9fe31
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-16T16:41:54Z

    moved some tests around and some other stuff

commit 6112b54e6e9e92fe8fb36c949237a7b418219aa4
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-16T17:28:44Z

    Took out the monitor thread, replaced with a schedule. The monitor thread because of the
Thread.sleep(#) was causing the shutdown behavior to be slow and masking issues that resulted
from improper handling of the queues.

commit e4725e6f5f78115a261e39f9d97e0fc43d5009e8
Author: Robert Douglas <rdouglas@w2odigital.com>
Date:   2014-05-16T19:18:03Z

    Added in additional delayed tests and removed one redundant LocalStreamBuilderTest

commit a3061d5c597e65217c6d18f03e58955f2b1ee451
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-16T19:21:03Z

    pushed more items down into the base class for simplicity and readability. Many things
are now marked as 'final' which shouldn't be changed during operation.

commit dd23158d824426047933a7ad8c6b3e76e3187823
Author: Robert Douglas <rdouglas@w2odigital.com>
Date:   2014-05-16T22:39:49Z

    Added tests to ensure that proper behavior occurs when errors are thrown at runtime in
various stages of the stream

commit 323d9c196af719605e2a55ce2572558ebcfbd98d
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-17T01:45:03Z

    All tests have been added, everything is working, many different delays, and use-cases
represented. All work and tests, save the delay tets, are extremely fast in the ms to produce
and shutdown properly.

commit 2a1928e1bb65d13c345c28537fb050eec6a398f9
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-16T19:21:03Z

    pushed more items down into the base class for simplicity and readability. Many things
are now marked as 'final' which shouldn't be changed during operation.

commit cbf223d7baf1106767d59218820b46dca748ed7b
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-17T01:45:03Z

    All tests have been added, everything is working, many different delays, and use-cases
represented. All work and tests, save the delay tets, are extremely fast in the ms to produce
and shutdown properly.

commit 40a258b21f641675233a8c84663bf2b7c7e5dd6b
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-17T03:50:48Z

    Merge branch 'robdouglas-Unit-Tests'
    
    Conflicts:
    	streams-runtimes/streams-runtime-local/src/test/java/org/apache/streams/local/builders/LocalStreamBuilderTest.java
    	streams-runtimes/streams-runtime-local/src/test/java/org/apache/streams/local/test/processors/PassThroughStaticCounterProcessor.java
    	streams-runtimes/streams-runtime-local/src/test/java/org/apache/streams/local/test/writer/DatumCounterWriter.java

commit aa2a0253f8a764b2687b8b4691af08e39322253d
Author: Matthew Hager <matthew.hager@gmail.com>
Date:   2014-05-17T04:45:22Z

    Completed overhaul of LocalStreamsbuilder. Much faster backwards compatible and now, complete
with over 30 working tests.

----


> Monitor thread pool never fully shuts down tasks in LocalStreamBuilder
> ----------------------------------------------------------------------
>
>                 Key: STREAMS-81
>                 URL: https://issues.apache.org/jira/browse/STREAMS-81
>             Project: Streams
>          Issue Type: Bug
>            Reporter: Robert Douglas
>
> In the current implementation of LocalStreamBuilder, the monitor thread pool never fully
shuts down all of its tasks.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message