streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Ebanks <ryaneba...@gmail.com>
Subject Re: LocalStreamBuilder#stop()
Date Wed, 08 Oct 2014 17:29:57 GMT
Ok.  I agree that 2b is the most appropriate.  It will require the most
code change though.  Right now the code does option 1 and test assumes
option 2.  I'll @Ignore the test for now, and we can make this change in
another pull request as I don't want this one to get too big. That means
the code will continue with option 1 for now.

-Ryan

On Wed, Oct 8, 2014 at 11:49 AM, Steve Blackmon <sblackmon@apache.org>
wrote:

> I also prefer 2b.
>
> The grace period before forceful termination should be configurable
> either at compile-time by the pipeline or preferably at run-time via
> the type safe config.
>
> A config fragment like this detected by the runtime and enforced
> absolutely would be a good start.  The implementation shouldn't be
> responsible for parsing args and creating a Map.
>
> runtime {
>   shutdown {
>     stopProviders {
>       timeAfterStart: "30m"
>     }
>     forceful {
>       timeAfterStopProviders: "5m"
>     }
>   }
> }
>
> Steve
> Steve Blackmon
> sblackmon@apache.org
>
>
> On Wed, Oct 8, 2014 at 10:55 AM, Robert Douglas
> <robert.baker.douglas@gmail.com> wrote:
> > Hi Ryan,
> >
> > I think the most important thing is to ensure that anytime "stop()" is
> > called, the behavior is deterministic. When that method is called, we
> need
> > a guarantee that things will completely shut down in a reasonable amount
> of
> > time. However, I don't like the idea of just abandoning datums that are
> > already in the pipeline. If it's possible, I'd prefer to go with an
> > approach similar to option 2b. Completely shutdown the providers, and
> give
> > the Stream some time to attempt to process all currently queued datums.
> > However, after a certain period of time, I think that a forceful shutdown
> > needs to be executed regardless of the state of the pipeline.
> >
> > -- Robert Douglas
> >
> > On Wed, Oct 8, 2014 at 10:31 AM, Ryan Ebanks <ryanebanks@gmail.com>
> wrote:
> >
> >> I've been added tests and fixing some issues in the local stream builder
> >> class.  I need some input on how the stop() method should work so I can
> >> write the appropriate tests.
> >>
> >> Option 1.) Gentle Shutdown
> >> Immediately stop the providers from producing more data, but allow for
> all
> >> data currently in the stream to finish processing.  Shutdown time would
> be
> >> unknown as a result.
> >>
> >> Option 2.) Forceful Shutdown
> >> Immediately stop the providers from producing more data, and stop all
> >> processors/writers.  Data will be left in the pipeline will be
> discarded.
> >> shutdown will be immediate
> >>
> >> Option 2b) Forceful Shutdown after small wait
> >> Immediately stop providers from producing more data.  Give the
> >> processors/writers a small window to try to finish processing (<
> 10secs),
> >> and if not finished, forcefully shutdown all processors/writers.
> >>
> >>
> >> These options can be changed in the future as more features are added.
> For
> >> example once datum tracking is implemented, we will probably want a
> >> combination of option 1 and 2b.  But we need a defined way for it work
> now,
> >> so tests are testing the correct functionality.
> >>
> >> -Ryan Ebanks
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message