reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashvin A <>
Subject Re: Non-blocking version
Date Mon, 20 Jun 2016 21:27:22 GMT

I am restarting this thread to verify if my observation is as expected.

As Markus suggested, I borrowed code to configure client and exit when a
long running job starts on YARN. I am invoking reef.close() after receiving
RunningJob event and then exit the function. Invoking reef.close on client
does not kill the running job. However, I see this log:
WARNING:  Force close job". This message is confusing as the job continues
to run on YARN. Should I ignore the log message or should I stop invoking


On Sat, Apr 9, 2016 at 8:17 AM, Markus Weimer <> wrote:

> Hi Ashvin,
> On 2016-04-08 17:12, Ashvin A wrote:
>> I am using to start a long running service.
> Uh oh, that thing is getting a life of its own :)
>  Is there a non-blocking version of this method which returns
>> without killing the service.
> Yes. DriverLauncher is actually a wrapper around our asynchronous API.
> That is the main API for the REEF Client, but it has proven to be quite
> hard to write integration tests and simple examples. Hence, we made
> `DriverLauncher` which offers a blocking call.
> If you want to make your own async client, the DriverLauncher source could
> be a good source of documentation. For example,
> `DriverLauncher.getLauncher()` contains the code to configure a client:
> ```
> final Configuration clientConfiguration = ClientConfiguration.CONF
>   .set(ClientConfiguration.ON_JOB_RUNNING, RunningJobHandler.class)
>   .set(ClientConfiguration.ON_JOB_COMPLETED, CompletedJobHandler.class)
>   .set(ClientConfiguration.ON_JOB_FAILED, FailedJobHandler.class)
>   .set(ClientConfiguration.ON_RUNTIME_ERROR, RuntimeErrorHandler.class)
>   .build();
> ```
> A REEF Client is very similar to a REEF Driver: It's a bunch of event
> handlers that REEF invokes. Once you fill out a `ClientConfiguration` like
> above, you merge it with the configuration of the runtime of choice and
> obtain a `REEF` instance. That instance can then be used to `.submit()` the
> DriverConfiguration to:
> ```
> Tang.Factory.getTang()
>   .newInjector(runtimeConfiguration, clientConfiguration)
>   .getInstance(REEF.class);
> ```
> `DriverLauncher` follows a slightly different pattern, where the
> `DriverLauncher` constructor gets the `REEF` instance. That has proven to
> be a useful pattern as it allows to pass other parameters besides the
> `REEF` instance.
> Please let me know how it goes. If the above works, we probably should
> turn it into a FAQ answer :)
> Markus

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message