flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: Including option for starting job and task managers in the foreground
Date Thu, 03 Dec 2015 09:55:55 GMT
I think the way supervisor is used in the Docker scripts is a bit hacky. It
is simply started in the foreground and does nothing. Supervisor is
actually a really nice utility to start processes in Docker containers and
monitor them.

Nevertheless, supervisor also expects commands to stay in the foreground. A
common way to work around this, is to create a script which monitors the
daemon process' pid. Thinking about this, I think we could actually add the
foreground functionality directly in the jobmanager / taskmanager shell
script like you suggested.

In the meantime, you could also use a simple script like this:

#!/usr/bin/env bash
# daemonize job manager
./bin/jobmanager start cluster
# wait until process goes down
wait $!

Cheers,
Max

On Wed, Dec 2, 2015 at 7:16 PM, Brian Chhun <brian.chhun@getbraintree.com>
wrote:

> Thanks, I'm basing the things I'm doing based on what I see there. One
> thing that's not clear to me in that example is why supervisor is used to
> keep the container alive, rather than using some simpler means. It doesn't
> look like it's been configured to supervise anything.
>
> On Wed, Dec 2, 2015 at 11:44 AM, Maximilian Michels <mxm@apache.org>
> wrote:
>
>> Have you looked at
>> https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
>> ? This demonstrates how to use Flink with Docker. In particular it
>> states: "Images [..] run Supervisor to stay alive when running
>> containers."
>>
>> Have a look at flink/config-flink.sh.
>>
>> Cheers,
>> Max
>>
>> On Wed, Dec 2, 2015 at 6:29 PM, Brian Chhun
>> <brian.chhun@getbraintree.com> wrote:
>> > Yep, I think this makes sense. I'm currently patching the
>> flink-daemon.sh
>> > script to remove the `&`, but I don't think it's a very robust solution,
>> > particularly when this script changes across versions of Flink. I'm
>> very new
>> > to Docker, but the resources I've found indicates that the process must
>> run
>> > in the foreground, though people seem to get around it with some hacks.
>> >
>> > When I have some time, I can look into refactoring some parts of the
>> scripts
>> > so that it can be started in the foreground.
>> >
>> > Thanks,
>> > Brian
>> >
>> > On Wed, Dec 2, 2015 at 3:22 AM, Maximilian Michels <mxm@apache.org>
>> wrote:
>> >>
>> >> Hi Brian,
>> >>
>> >> I don't recall Docker requires commands to run in the foreground.
>> Still,
>> >> if that is your requirement, simply remove the "&" at the end of this
>> line
>> >> in flink-daemon.sh:
>> >>
>> >> $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}"
>> -classpath
>> >> "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
>> >> ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
>> >>
>> >> Cheers,
>> >> Max
>> >>
>> >> On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <trohrmann@apache.org>
>> >> wrote:
>> >>>
>> >>> Hi Brian,
>> >>>
>> >>> as far as I know this is at the moment not possible with our scripts.
>> >>> However it should be relatively easy to add by simply executing the
>> Java
>> >>> command in flink-daemon.sh in the foreground. Do you want to add this?
>> >>>
>> >>> Cheers,
>> >>> Till
>> >>>
>> >>> On Dec 1, 2015 9:40 PM, "Brian Chhun" <brian.chhun@getbraintree.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi All,
>> >>>>
>> >>>> Is it possible to include a command line flag for starting job and
>> task
>> >>>> managers in the foreground? Currently, `bin/jobmanager.sh` and
>> >>>> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts
>> these
>> >>>> things in the background. I'd like to execute these commands inside
>> a docker
>> >>>> container, but it's expected that the process is running in the
>> foreground.
>> >>>> I think it might be useful to have it run in the foreground so that
>> it can
>> >>>> be hooked into some process supervisors. Any suggestions are
>> appreciated.
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>> Brian
>> >>
>> >>
>> >
>>
>
>

Mime
View raw message