flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Wu <stevenz...@gmail.com>
Subject Re: entrypoint for executing job in task manager
Date Thu, 21 Dec 2017 16:49:52 GMT
We use Guice for dependency injection. We need to install *additional*
Guice modules (for bindings) when setting up this static context of Guice

Calling the static initializer from operator open method won't really help.
Not all operators are implemented by app developer who want to install
additional Guice modules. E.g. kafka source operator is
implemented/provided by our platform. I think the source operator will open
first, which means app operator won't get a chance to initialize the static
context. What would really help if there is a entry hook (at task manager)
that is executed before any operator opening.

On Thu, Dec 21, 2017 at 12:27 AM, Piotr Nowojski <piotr@data-artisans.com>

> Open method is called just before any elements are processed. You can hook
> in any initialisation logic there, including initialisation of a static
> context. However keep in mind, that since this context is static, it will
> be shared between multiple operators (if you are running parallelism >
> number of task managers), so accesses to it must be synchronized (including
> initialisation). Another thing to consider is that managing the life cycle
> of static context can be tricky (when to close it and release it’s
> resources).
> The questions is, whether you really need a static context?
> Thanks,
> Piotrek
> > On 21 Dec 2017, at 07:53, Steven Wu <stevenz3wu@gmail.com> wrote:
> >
> > Here is my understanding of how job submission works in Flink. When
> submitting a job to job manager via REST API, we provide a entry class. Job
> manager then evaluate job graph and ship serialized operators to task
> manager. Task manager then open operators and run tasks.
> >
> > My app would typically requires some initialization phase to setup my
> own running context in task manager (e.g. calling a static method of some
> class). Does Flink provide any entry hook in task manager when executing a
> job (and tasks)? As for job manager, the entry class provides such hook
> where I can initialize my static context.
> >
> > Thanks,
> > Steven

View raw message