Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@streams.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of ryanebanks@gmail.com designates
 209.85.218.43 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAJ7ACC_GCN3P_vN=go9G-RiANGEO5P9wSZSHX_qegMBJiXWbJQ@mail.gmail.com>
References: 
 <CAK8vTf9BDxSYjPjVNzYivgQq4Lwhnbn0fGgU5xZwWco1smm30g@mail.gmail.com>
	<CAHFTPLSN9fjwsvm4m5B=_pqqRp4FwHX0+PbFeRB+9FCE2CdMUQ@mail.gmail.com>
	<CAJ7ACC_GCN3P_vN=go9G-RiANGEO5P9wSZSHX_qegMBJiXWbJQ@mail.gmail.com>
Date: Wed, 8 Oct 2014 12:29:57 -0500
Message-ID: 
 <CAK8vTf8RnyC2CHV3VkVO9D+rZXYqeC-sNRHwOseJm_ObFmCiRg@mail.gmail.com>
Subject: Re: LocalStreamBuilder#stop()
From: Ryan Ebanks <ryanebanks@gmail.com>
To: dev@streams.incubator.apache.org
Content-Type: multipart/alternative; boundary=047d7b5d375a2d38ee0504ecaa58

--047d7b5d375a2d38ee0504ecaa58
Content-Type: text/plain; charset=UTF-8

Ok.  I agree that 2b is the most appropriate.  It will require the most
code change though.  Right now the code does option 1 and test assumes
option 2.  I'll @Ignore the test for now, and we can make this change in
another pull request as I don't want this one to get too big. That means
the code will continue with option 1 for now.

-Ryan

On Wed, Oct 8, 2014 at 11:49 AM, Steve Blackmon <sblackmon@apache.org>
wrote:

> I also prefer 2b.
>
> The grace period before forceful termination should be configurable
> either at compile-time by the pipeline or preferably at run-time via
> the type safe config.
>
> A config fragment like this detected by the runtime and enforced
> absolutely would be a good start.  The implementation shouldn't be
> responsible for parsing args and creating a Map.
>
> runtime {
>   shutdown {
>     stopProviders {
>       timeAfterStart: "30m"
>     }
>     forceful {
>       timeAfterStopProviders: "5m"
>     }
>   }
> }
>
> Steve
> Steve Blackmon
> sblackmon@apache.org
>
>
> On Wed, Oct 8, 2014 at 10:55 AM, Robert Douglas
> <robert.baker.douglas@gmail.com> wrote:
> > Hi Ryan,
> >
> > I think the most important thing is to ensure that anytime "stop()" is
> > called, the behavior is deterministic. When that method is called, we
> need
> > a guarantee that things will completely shut down in a reasonable amount
> of
> > time. However, I don't like the idea of just abandoning datums that are
> > already in the pipeline. If it's possible, I'd prefer to go with an
> > approach similar to option 2b. Completely shutdown the providers, and
> give
> > the Stream some time to attempt to process all currently queued datums.
> > However, after a certain period of time, I think that a forceful shutdown
> > needs to be executed regardless of the state of the pipeline.
> >
> > -- Robert Douglas
> >
> > On Wed, Oct 8, 2014 at 10:31 AM, Ryan Ebanks <ryanebanks@gmail.com>
> wrote:
> >
> >> I've been added tests and fixing some issues in the local stream builder
> >> class.  I need some input on how the stop() method should work so I can
> >> write the appropriate tests.
> >>
> >> Option 1.) Gentle Shutdown
> >> Immediately stop the providers from producing more data, but allow for
> all
> >> data currently in the stream to finish processing.  Shutdown time would
> be
> >> unknown as a result.
> >>
> >> Option 2.) Forceful Shutdown
> >> Immediately stop the providers from producing more data, and stop all
> >> processors/writers.  Data will be left in the pipeline will be
> discarded.
> >> shutdown will be immediate
> >>
> >> Option 2b) Forceful Shutdown after small wait
> >> Immediately stop providers from producing more data.  Give the
> >> processors/writers a small window to try to finish processing (<
> 10secs),
> >> and if not finished, forcefully shutdown all processors/writers.
> >>
> >>
> >> These options can be changed in the future as more features are added.
> For
> >> example once datum tracking is implemented, we will probably want a
> >> combination of option 1 and 2b.  But we need a defined way for it work
> now,
> >> so tests are testing the correct functionality.
> >>
> >> -Ryan Ebanks
> >>
>

--047d7b5d375a2d38ee0504ecaa58--