Mailing-List: contact dev-help@sqoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@sqoop.apache.org
Date: Wed, 22 Oct 2014 18:04:34 +0000 (UTC)
From: "Jarek Jarcec Cecho (JIRA)" <jira@apache.org>
To: dev@sqoop.apache.org
Message-ID: <JIRA.12749828.1413998470000.316203.1414001074078@Atlassian.JIRA>
In-Reply-To: <JIRA.12749828.1413998470000@Atlassian.JIRA>
References: <JIRA.12749828.1413998470000@Atlassian.JIRA>
 <JIRA.12749828.1413998470652@arcas>
Subject: [jira] [Commented] (SQOOP-1603)  Sqoop2:  Explicit support for
 Merge in the Sqoop Job lifecyle
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SQOOP-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180255#comment-14180255 ] 

Jarek Jarcec Cecho commented on SQOOP-1603:
-------------------------------------------

I guess that you can blame me for this one for choosing confusing name :) Even thought that we're calling it a "destroyer" it's modeled after MapReduce's OutputCommitter, where you are enabled to do any "finish" work for the transfer. "Committing data", moving them from temporary directories to final one are expected operations.

>  Sqoop2:  Explicit support for Merge in the Sqoop Job lifecyle
> --------------------------------------------------------------
>
>                 Key: SQOOP-1603
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1603
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Veena Basavaraj
>            Assignee: Qian Xu
>
> This ticket was created while reviewing the Kite Connector use case where the destroyer does the actual temp data set merge
> https://reviews.apache.org/r/26963/diff/# [~stanleyxu2005]
> {code}
> public void destroy(DestroyerContext context, LinkConfiguration link,
>       ToJobConfiguration job) {
>     LOG.info("Running Kite connector destroyer");
>     // Every loader instance creates a temporary dataset. If the MR job is
>     // successful, all temporary dataset should be merged as one dataset,
>     // otherwise they should be deleted all.
>     String[] uris = KiteDatasetExecutor.listTemporaryDatasetUris(
>         job.toDataset.uri);
>     if (context.isSuccess()) {
>       KiteDatasetExecutor executor = new KiteDatasetExecutor(job.toDataset.uri,
>           context.getSchema(), link.link.fileFormat);
>       for (String uri : uris) {
>         executor.mergeDataset(uri);
>         LOG.info(String.format("Temporary dataset %s merged", uri));
>       }
>     } else {
>       for (String uri : uris) {
>         KiteDatasetExecutor.deleteDataset(uri);
>         LOG.info(String.format("Temporary dataset %s deleted", uri));
>       }
>     }
>   }
> {code}
> Wondering if such things should be its own phase rather than in destroyers. The responsibility of destroyer is more to clean up/ closing/ anything thats pretty much destroying, should such operations that modify records its own step.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)