flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <till.rohrm...@gmail.com>
Subject Re: Multiple restarts of Local Cluster
Date Wed, 02 Sep 2015 16:22:11 GMT
If I'm not mistaken, then the cluster should be properly terminated when it
gets garbage collected. Thus, also when the main method exits.

On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel <sachingoel0101@gmail.com>
wrote:

> If I'm right, all Tests use either the MultipleProgramTestBase or
> JavaProgramTestBase​. Those shut down the cluster explicitly anyway.
> I will make sure if this is the case.
>
> Regards
> Sachin
>
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>
> On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann <trohrmann@apache.org>
> wrote:
>
>> Maybe we can create a single PlanExecutor for the LocalEnvironment which
>> is used when calling execute. This of course entails that we don’t call
>> stop on the LocalCluster. For cases where the program exits after
>> calling execute, this should be fine because all resources will then be
>> released anyway. It might matter for the test execution where maven reuses
>> the JVMs and where the LocalFlinkMiniCluster won’t be garbage collected
>> right away. You could try it out and see what happens.
>>
>> Cheers,
>> Till
>> ​
>>
>> On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann <trohrmann@apache.org>
>> wrote:
>>
>>> Oh sorry, then I got the wrong context. I somehow thought it was about
>>> test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.
>>>
>>> On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <sachingoel0101@gmail.com>
>>> wrote:
>>>
>>>> I was under the impression that the @AfterClass annotation can only be
>>>> used in test classes.
>>>> Even so, the idea is that a user program running in the IDE should not
>>>> be starting up the cluster several times [my primary concern is the
>>>> addition of the persist operator], and we certainly cannot ask the user to
>>>> terminate the cluster after execution, while in local mode.
>>>>
>>>> -- Sachin Goel
>>>> Computer Science, IIT Delhi
>>>> m. +91-9871457685
>>>>
>>>> On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann <trohrmann@apache.org>
>>>> wrote:
>>>>
>>>>> Why is it not possible to shut down the local cluster? Can’t you shut
>>>>> it down in the @AfterClass method?
>>>>> ​
>>>>>
>>>>> On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <sachingoel0101@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes. That will work too. However, then it isn't possible to shut
down
>>>>>> the local cluster. [Is it necessary to do so or does it shut down
>>>>>> automatically when the program exists? I'm not entirely sure.]
>>>>>>
>>>>>> -- Sachin Goel
>>>>>> Computer Science, IIT Delhi
>>>>>> m. +91-9871457685
>>>>>>
>>>>>> On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen <sewen@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Have a look at some other tests, like the checkpointing tests.
They
>>>>>>> start one cluster manually and keep it running. They connect
against it
>>>>>>> using the remote environment ("localhost",
>>>>>>> miniCluster.getJobManagerRpcPort()).
>>>>>>>
>>>>>>> That works nicely...
>>>>>>>
>>>>>>> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <
>>>>>>> sachingoel0101@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi all
>>>>>>>> While using LocalEnvironment, in case the program triggers
>>>>>>>> execution several times, the {{LocalFlinkMiniCluster}} is
started as many
>>>>>>>> times. This can consume a lot of time in setting up and tearing
down the
>>>>>>>> cluster. Further, this hinders with a new functionality I'm
working on
>>>>>>>> based on persisted results.
>>>>>>>> One potential solution could be to follow the methodology
in
>>>>>>>> `MultipleProgramsTestBase`. The user code then would have
to reside in a
>>>>>>>> fixed function name, instead of the main method. Or is that
too cumbersome?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Sachin
>>>>>>>> -- Sachin Goel
>>>>>>>> Computer Science, IIT Delhi
>>>>>>>> m. +91-9871457685
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message