falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-107) Adding extensions
Date Mon, 16 Sep 2013 06:58:56 GMT

    [ https://issues.apache.org/jira/browse/FALCON-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768112#comment-13768112

Srikanth Sundarrajan commented on FALCON-107:

Perhaps we should call this path.libext....., as they are paths and it is better to be explicit

It might make sense to consider other non-standard actions which might similarly include lib
files such as mr/pig/java. 
+            for(Object obj: wf.getDecisionOrForkOrJoin()) {
+                if(!(obj instanceof ACTION))
+                    continue;
+                ACTION action = (ACTION) obj;
+                List<String> files = null;
+                if (action.getJava() != null) {
+                    files = action.getJava().getFile();
+                } else if (action.getPig() != null) {
+                    files = action.getPig().getFile();
+                } else if(action.getMapReduce() != null) {
+                    files = action.getMapReduce().getFile();
+                }
+                if (files != null) {
+                    files.add(lib.getPath().toString());
+                }
+            }

Signature can be modified to source,target paths instead of local & cluster. That would
be more explicit and be more descriptive.
+    public static void pushLibsToHDFS(String localPaths, Path clusterPath, Cluster cluster,
FalconPathFilter pathFilter)

Increasingly there is case for lifecycle to be handled explicitly. [~svenkat] was attempting
to add this as part of FALCON-36, perhaps it might be better to expedite this, as work on
that has been parked right now. I would like schedulable entities and their lifecycles to
be enumerated and libs pushed. May be we can move this to a separate JIRA.
+            pushLibsToHDFS(properties.getProperty("libext"), libext, cluster, null);
+            pushLibsToHDFS(properties.getProperty("libext.feed"), new Path(libext, EntityType.FEED.name())
, cluster, null);
+            pushLibsToHDFS(properties.getProperty("libext.feed.replication"), new Path(libext,
EntityType.FEED.name() + "/replication"), cluster, null);
+            pushLibsToHDFS(properties.getProperty("libext.feed.retention"), new Path(libext,
EntityType.FEED.name() + "/retention"), cluster, null);
+            pushLibsToHDFS(properties.getProperty("libext.process"), new Path(libext, EntityType.PROCESS.name())
, cluster, null);

* Would like to see an assertion in EntityManagerJerseyIT to ensure that extension libs are
indeed copied to cluster post submit
* More tests (of Mappers) to ensure that extension libs are indeed included in workflows would
be nice.
> Adding extensions
> -----------------
>                 Key: FALCON-107
>                 URL: https://issues.apache.org/jira/browse/FALCON-107
>             Project: Falcon
>          Issue Type: New Feature
>            Reporter: Shwetha G S
>            Assignee: Shwetha G S
>             Fix For: 0.4
>         Attachments: FALCON-107-v1.patch
> Falcon should support ability to add extensions like new filesystems(if not part of hadoop
release). In general, to add extra libraries that are available for falcon service, and feed/process

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message