flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Nowojski <pi...@data-artisans.com>
Subject Re: entrypoint for executing job in task manager
Date Fri, 22 Dec 2017 10:23:31 GMT
I don’t think there is such hook in the Flink code now. You will have to walk around this
issue somehow in user space. 

Maybe you could make a contract that every operator before touching Guice, should call static
synchronized method `initializeGuiceContext`. This method could search the classpath for classes
with some specific annotations, for example `@MyInitializationHook` and install/add all of
such hooks before actually using Guice?

Piotrek

> On 21 Dec 2017, at 17:49, Steven Wu <stevenz3wu@gmail.com> wrote:
> 
> We use Guice for dependency injection. We need to install additional Guice modules (for
bindings) when setting up this static context of Guice injector.
> 
> Calling the static initializer from operator open method won't really help. Not all operators
are implemented by app developer who want to install additional Guice modules. E.g. kafka
source operator is implemented/provided by our platform. I think the source operator will
open first, which means app operator won't get a chance to initialize the static context.
What would really help if there is a entry hook (at task manager) that is executed before
any operator opening.
> 
> On Thu, Dec 21, 2017 at 12:27 AM, Piotr Nowojski <piotr@data-artisans.com <mailto:piotr@data-artisans.com>>
wrote:
> Open method is called just before any elements are processed. You can hook in any initialisation
logic there, including initialisation of a static context. However keep in mind, that since
this context is static, it will be shared between multiple operators (if you are running parallelism
> number of task managers), so accesses to it must be synchronized (including initialisation).
Another thing to consider is that managing the life cycle of static context can be tricky
(when to close it and release it’s resources).
> 
> The questions is, whether you really need a static context?
> 
> Thanks,
> Piotrek
> 
> 
> > On 21 Dec 2017, at 07:53, Steven Wu <stevenz3wu@gmail.com <mailto:stevenz3wu@gmail.com>>
wrote:
> >
> > Here is my understanding of how job submission works in Flink. When submitting a
job to job manager via REST API, we provide a entry class. Job manager then evaluate job graph
and ship serialized operators to task manager. Task manager then open operators and run tasks.
> >
> > My app would typically requires some initialization phase to setup my own running
context in task manager (e.g. calling a static method of some class). Does Flink provide any
entry hook in task manager when executing a job (and tasks)? As for job manager, the entry
class provides such hook where I can initialize my static context.
> >
> > Thanks,
> > Steven
> 
> 


Mime
View raw message