falcon-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From peeyu...@apache.org
Subject falcon git commit: FALCON-1980 Change input and output argument order for Spark process workflow
Date Mon, 30 May 2016 05:16:42 GMT
Repository: falcon
Updated Branches:
  refs/heads/0.10 16a07cb31 -> 33a37a638


FALCON-1980 Change input and output argument order for Spark process workflow

Details to perform this change has been mentioned in the FALCON-1980.

Author: peeyush b <pbishnoi@hortonworks.com>

Reviewers: Pavan <pavan.kolamuri@gmail.com>

Closes #160 from peeyushb/FALCON-1980 and squashes the following commits:

23d4e24 [peeyush b] Added comments for input and output argument
3d86f58 [peeyush b] FALCON-1980 : Change input and output argument order for Spark process
workflow

(cherry picked from commit 400ef329d1816b408e759990b880c9dd1970dae2)
Signed-off-by: peeyush b <pbishnoi@hortonworks.com>


Project: http://git-wip-us.apache.org/repos/asf/falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/falcon/commit/33a37a63
Tree: http://git-wip-us.apache.org/repos/asf/falcon/tree/33a37a63
Diff: http://git-wip-us.apache.org/repos/asf/falcon/diff/33a37a63

Branch: refs/heads/0.10
Commit: 33a37a6380f16830265d23c47d444cd3b1d7ddbe
Parents: 16a07cb
Author: Peeyush <peeyushb@apache.org>
Authored: Mon May 30 10:46:02 2016 +0530
Committer: peeyush b <pbishnoi@hortonworks.com>
Committed: Mon May 30 10:46:28 2016 +0530

----------------------------------------------------------------------
 .../oozie/process/SparkProcessWorkflowBuilder.java      | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/falcon/blob/33a37a63/oozie/src/main/java/org/apache/falcon/oozie/process/SparkProcessWorkflowBuilder.java
----------------------------------------------------------------------
diff --git a/oozie/src/main/java/org/apache/falcon/oozie/process/SparkProcessWorkflowBuilder.java
b/oozie/src/main/java/org/apache/falcon/oozie/process/SparkProcessWorkflowBuilder.java
index dc5a491..8c06711 100644
--- a/oozie/src/main/java/org/apache/falcon/oozie/process/SparkProcessWorkflowBuilder.java
+++ b/oozie/src/main/java/org/apache/falcon/oozie/process/SparkProcessWorkflowBuilder.java
@@ -89,8 +89,9 @@ public class SparkProcessWorkflowBuilder extends ProcessExecutionWorkflowBuilder
             argList.addAll(sparkArgs);
         }
 
-        addInputFeedsAsArgument(argList, cluster);
+        //Adding output first so that final order must have input and then output followed
by user's arguments.
         addOutputFeedsAsArgument(argList, cluster);
+        addInputFeedsAsArgument(argList, cluster);
 
         sparkAction.setJar(addUri(sparkFilePath, cluster));
 
@@ -145,6 +146,7 @@ public class SparkProcessWorkflowBuilder extends ProcessExecutionWorkflowBuilder
             return;
         }
 
+        //Adding to the 0th index and getting the args shifted as arguments are added to
get the desired effect.
         int numInputFeed = entity.getInputs().getInputs().size();
         while (numInputFeed > 0) {
             Input input = entity.getInputs().getInputs().get(numInputFeed-1);
@@ -163,13 +165,17 @@ public class SparkProcessWorkflowBuilder extends ProcessExecutionWorkflowBuilder
             return;
         }
 
-        for(Output output : entity.getOutputs().getOutputs()) {
+        //Adding to the 0th index and getting the args shifted as arguments are added to
get the desired effect.
+        int numOutputFeed = entity.getOutputs().getOutputs().size();
+        while (numOutputFeed > 0) {
+            Output output = entity.getOutputs().getOutputs().get(numOutputFeed-1);
             Feed feed = EntityUtil.getEntity(EntityType.FEED, output.getFeed());
             Storage storage = FeedHelper.createStorage(cluster, feed);
             final String outputName = output.getName();
             if (storage.getType() == Storage.TYPE.FILESYSTEM) {
-                argList.add(argList.size(), "${" + outputName + "}");
+                argList.add(0, "${" + outputName + "}");
             }
+            numOutputFeed--;
         }
     }
 


Mime
View raw message