streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Blackmon <sblack...@apache.org>
Subject Re: LocalStreamBuilder#stop()
Date Wed, 08 Oct 2014 16:49:23 GMT
I also prefer 2b.

The grace period before forceful termination should be configurable
either at compile-time by the pipeline or preferably at run-time via
the type safe config.

A config fragment like this detected by the runtime and enforced
absolutely would be a good start.  The implementation shouldn't be
responsible for parsing args and creating a Map.

runtime {
  shutdown {
    stopProviders {
      timeAfterStart: "30m"
    }
    forceful {
      timeAfterStopProviders: "5m"
    }
  }
}

Steve
Steve Blackmon
sblackmon@apache.org


On Wed, Oct 8, 2014 at 10:55 AM, Robert Douglas
<robert.baker.douglas@gmail.com> wrote:
> Hi Ryan,
>
> I think the most important thing is to ensure that anytime "stop()" is
> called, the behavior is deterministic. When that method is called, we need
> a guarantee that things will completely shut down in a reasonable amount of
> time. However, I don't like the idea of just abandoning datums that are
> already in the pipeline. If it's possible, I'd prefer to go with an
> approach similar to option 2b. Completely shutdown the providers, and give
> the Stream some time to attempt to process all currently queued datums.
> However, after a certain period of time, I think that a forceful shutdown
> needs to be executed regardless of the state of the pipeline.
>
> -- Robert Douglas
>
> On Wed, Oct 8, 2014 at 10:31 AM, Ryan Ebanks <ryanebanks@gmail.com> wrote:
>
>> I've been added tests and fixing some issues in the local stream builder
>> class.  I need some input on how the stop() method should work so I can
>> write the appropriate tests.
>>
>> Option 1.) Gentle Shutdown
>> Immediately stop the providers from producing more data, but allow for all
>> data currently in the stream to finish processing.  Shutdown time would be
>> unknown as a result.
>>
>> Option 2.) Forceful Shutdown
>> Immediately stop the providers from producing more data, and stop all
>> processors/writers.  Data will be left in the pipeline will be discarded.
>> shutdown will be immediate
>>
>> Option 2b) Forceful Shutdown after small wait
>> Immediately stop providers from producing more data.  Give the
>> processors/writers a small window to try to finish processing (< 10secs),
>> and if not finished, forcefully shutdown all processors/writers.
>>
>>
>> These options can be changed in the future as more features are added.  For
>> example once datum tracking is implemented, we will probably want a
>> combination of option 1 and 2b.  But we need a defined way for it work now,
>> so tests are testing the correct functionality.
>>
>> -Ryan Ebanks
>>

Mime
View raw message