falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1105) Recipe repository management API's and CLI support
Date Thu, 19 Mar 2015 19:35:38 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369975#comment-14369975
] 

Ajay Yadava commented on FALCON-1105:
-------------------------------------

[~sowmyaramesh] Seems like a non-trivial patch, can you please add it to reviewboard?

Also, irrespective of where the resources go (depending on the choice we make in FALCON-1108),
CLI side of recipe code should be separated from existing one as it's turning into one large
monolithic piece of code. You can refer FalconMetaDataCLI for reference.

Will review it completely once you upload the patch to reviewboard.

> Recipe repository management API's and CLI support
> --------------------------------------------------
>
>                 Key: FALCON-1105
>                 URL: https://issues.apache.org/jira/browse/FALCON-1105
>             Project: Falcon
>          Issue Type: Sub-task
>          Components: client
>    Affects Versions: 0.6
>            Reporter: Sowmya Ramesh
>            Assignee: Sowmya Ramesh
>              Labels: Recipe
>             Fix For: 0.7
>
>         Attachments: FALCON-1105.v0.patch
>
>
> *Recipe Listing*
> A GET method may be added to the Prism server on a new Jersey resource to list recipes
and their corresponding root location on recipe repository. Corresponding CLI methods to be
present
> {noformat}
> [hrt_qa@node-1 recipe]$ falcon recipe -list
> List of recipes
> {
>   "totalSize": 2,
>   "results": {
>     "hdfs-replication": "hdfs://node-1.example.com:8020/apps/falcon/recipe/hdfs-replication",
>     "hive-disaster-recovery": "hdfs://node-1.example.com:8020/apps/falcon/recipe/hive-disaster-recovery"
>   }
> }
> {noformat}
> *Get resources of a given recipe*
> List all the resources of a given recipe
> {noformat}
> [hrt_qa@node-1 recipe]$ falcon recipe -getResources -name hdfs-replication
> {
>   "totalSize": 3,
>   "results": {
>     "hdfs-replication-template.xml": "hdfs://node-1.example.com:8020/apps/falcon/recipe/hdfs-replication/resources/build/hdfs-replication-template.xml",
>     "hdfs-replication-workflow.xml": "hdfs://node-1.example.com:8020/apps/falcon/recipe/hdfs-replication/resources/runtime/hdfs-replication-workflow.xml",
>     "hdfs-replication.properties": "hdfs://node-1.example.com:8020/apps/falcon/recipe/hdfs-replication/resources/build/hdfs-replication.properties"
>   }
> }
> {noformat}
> *Recipe Description*
> A GET method may be added to the Prism server to echo the README as documentation for
the users. This may contain brief on the functionality offered by the recipe and any operability
notes of importance
> {noformat}
> [hrt_qa@node-1 recipe]$ falcon recipe -describe -name hdfs-replication
> HDFS Directory Replication Recipe
> Overview
> This recipe implements replicating arbitrary directories on HDFS from one
> Hadoop cluster to another Hadoop cluster.
> This piggy backs on replication solution in Falcon which uses the DistCp tool.
> Use Case
> * Copy directories between HDFS clusters with out dated partitions
> * Archive directories from HDFS to Cloud. Ex: S3, Azure WASB
> Limitations
> As the data volume and number of files grow, this can get inefficient.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message