falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-369) Refactor workflow builder
Date Mon, 16 Jun 2014 14:50:02 GMT

    [ https://issues.apache.org/jira/browse/FALCON-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032492#comment-14032492
] 

Srikanth Sundarrajan commented on FALCON-369:
---------------------------------------------

Patch doesn't seem to apply cleanly. Can this be rebased
{code}
Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java...
Hunk #2 succeeded at 43 (offset 1 line).
error: while searching for:
        }

        if (!schedClusters.isEmpty()) {
            WorkflowBuilder<Entity> builder = WorkflowBuilder.getBuilder(ENGINE, entity);
            Map<String, Properties> newFlows = builder.newWorkflowSchedule(schedClusters.toArray(new
                String[schedClusters.size()]));
            for (Map.Entry<String, Properties> entry : newFlows.entrySet()) {
                String cluster = entry.getKey();
                LOG.info("Scheduling {} on cluster {}", entity.toShortString(), cluster);
                scheduleEntity(cluster, entry.getValue(), entity);
                commitStagingPath(cluster, entry.getValue().getProperty(OozieClient.BUNDLE_APP_PATH));
            }
        }
    }

    private void commitStagingPath(String cluster, String path) throws FalconException {
        path = StringUtils.removeStart(path, "${nameNode}");
        Cluster clusterEntity = ConfigurationStore.get().get(EntityType.CLUSTER, cluster);
        FileSystem fs = HadoopClientFactory.get().createFileSystem(ClusterHelper.getConfiguration(clusterEntity));
        try {
            fs.create(new Path(path, EntityUtil.SUCCEEDED_FILE_NAME)).close();
        } catch (IOException e) {

error: patch failed: oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java:139
error: oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java: patch
does not apply
{code}

> Refactor workflow builder
> -------------------------
>
>                 Key: FALCON-369
>                 URL: https://issues.apache.org/jira/browse/FALCON-369
>             Project: Falcon
>          Issue Type: Improvement
>            Reporter: Shwetha G S
>            Assignee: Shwetha G S
>         Attachments: FALCON-369.patch, FalconWorkflowBuilder.png
>
>
> Currently, feed/process workflow builder is a single class which handles all different
cases of lifecycles, storage types, workflow engines and building all oozie entities(workflow,
coord and bundle). This is not readable and difficult to maintain. This needs some re-factoring.
> Approach:
> Maintain different builders for
> 1. oozie entities - workflow, coord and bundle. 
> 2. entity types - feed and process
> 3. lifecycle - process, retention and replication
> 4. workflow engines - oozie, pig and hive



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message