sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Jarcec Cecho (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1638) Notification on Submission.
Date Wed, 29 Oct 2014 15:23:33 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188443#comment-14188443

Jarek Jarcec Cecho commented on SQOOP-1638:

We have a loop in [JobManager|https://github.com/apache/sqoop/blob/sqoop2/core/src/main/java/org/apache/sqoop/driver/JobManager.java#L669]
that is periodically checking status of currently running jobs and updating their status.
The problem with that loop is that it do have a delay - it's configurable but yet if we check
the job's status every 5 minutes, we can end up having up to five minutes of delay where Sqoop
thinks that the job is still running when it's not.

Hadoop do have ability to send a HTTP request to given configured URL when a job has been
finished (both successfully or on failure) - it's sort of a notification mechanism. I'm not
sure whether other execution engines (namely spark) do have this ability, but I would expect
so. We're using this ability to cut the delay on update loop. Once Hadoop execution is done,
we get called on the notification end point, we will get the latest status (hence the call
to {{status()}}) and persist it in our repository. As Hadoop doesn't care about return value,
we are returning {{EMPTY_BEAN}} back.

Obviously this end point is more an internal of Sqoop 2 that is exposed to execution engine
rather then endpoint that is meant to be used by a user.

> Notification on Submission.
> ---------------------------
>                 Key: SQOOP-1638
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1638
>             Project: Sqoop
>          Issue Type: Sub-task
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: 2.0.0
> What is this really supposed to do?
> There was some hacky code to set this URL
> {code}
> // TODO: This should be outsourced somewhere more suitable than here
>         if(JobManager.getInstance().getNotificationBaseUrl() == null) {
>           String url = ctx.getRequest().getRequestURL().toString();
>           JobManager.getInstance().setNotificationBaseUrl(
>             url.split("v1")[0] + "/v1/submission/notification/");
>         }
> {code}
> But all it did was return status
> {code}
>   private JsonBean handleNotification(RequestContext ctx, String sjid) {
>     LOG.debug("Received notification request for job " + sjid);
>     JobManager.getInstance().status(Long.parseLong(sjid));
>     return JsonBean.EMPTY_BEAN;
>   }
> {code}

This message was sent by Atlassian JIRA

View raw message