Return-Path: X-Original-To: apmail-falcon-commits-archive@minotaur.apache.org Delivered-To: apmail-falcon-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 08F4218823 for ; Mon, 15 Feb 2016 05:48:11 +0000 (UTC) Received: (qmail 75762 invoked by uid 500); 15 Feb 2016 05:48:10 -0000 Delivered-To: apmail-falcon-commits-archive@falcon.apache.org Received: (qmail 75727 invoked by uid 500); 15 Feb 2016 05:48:10 -0000 Mailing-List: contact commits-help@falcon.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.apache.org Delivered-To: mailing list commits@falcon.apache.org Received: (qmail 75718 invoked by uid 99); 15 Feb 2016 05:48:10 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Feb 2016 05:48:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id EE3EBC0BF2 for ; Mon, 15 Feb 2016 05:48:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.795 X-Spam-Level: * X-Spam-Status: No, score=1.795 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-0.006, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id R8ecMJC3lbJB for ; Mon, 15 Feb 2016 05:48:04 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTP id 920CD31AFB for ; Mon, 15 Feb 2016 05:48:03 +0000 (UTC) Received: from svn01-us-west.apache.org (svn.apache.org [10.41.0.6]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 9C4C2E0FD1 for ; Mon, 15 Feb 2016 05:48:02 +0000 (UTC) Received: from svn01-us-west.apache.org (localhost [127.0.0.1]) by svn01-us-west.apache.org (ASF Mail Server at svn01-us-west.apache.org) with ESMTP id 7D8843A10A5 for ; Mon, 15 Feb 2016 05:48:02 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: svn commit: r1730449 [3/3] - in /falcon/trunk: ./ general/ general/src/site/ general/src/site/twiki/ general/src/site/twiki/falconcli/ general/src/site/twiki/restapi/ releases/ Date: Mon, 15 Feb 2016 05:48:01 -0000 To: commits@falcon.apache.org From: pallavi@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20160215054802.7D8843A10A5@svn01-us-west.apache.org> Added: falcon/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki?rev=1730449&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki (added) +++ falcon/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki Mon Feb 15 05:48:00 2016 @@ -0,0 +1,14 @@ +---+++Update + +[[CommonCLI][Common CLI Options]] + +Update operation allows an already submitted/scheduled entity to be updated. Cluster and datasource updates are +currently not allowed. + +Usage: +$FALCON_HOME/bin/falcon entity -type [feed|process] -name <> -update -file <> + +Optional Arg : -skipDryRun. When this argument is specified, Falcon skips oozie dryrun. + +Example: +$FALCON_HOME/bin/falcon entity -type process -name hourly-reports-generator -update -file /process/definition.xml Added: falcon/trunk/general/src/site/twiki/falconcli/VersionAdmin.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/falconcli/VersionAdmin.twiki?rev=1730449&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/falconcli/VersionAdmin.twiki (added) +++ falcon/trunk/general/src/site/twiki/falconcli/VersionAdmin.twiki Mon Feb 15 05:48:00 2016 @@ -0,0 +1,7 @@ +---+++Version + +[[CommonCLI][Common CLI Options]] + +Version returns the current version of Falcon installed. +Usage: +$FALCON_HOME/bin/falcon admin -version Added: falcon/trunk/general/src/site/twiki/falconcli/VertexEdgesMetadata.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/falconcli/VertexEdgesMetadata.twiki?rev=1730449&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/falconcli/VertexEdgesMetadata.twiki (added) +++ falcon/trunk/general/src/site/twiki/falconcli/VertexEdgesMetadata.twiki Mon Feb 15 05:48:00 2016 @@ -0,0 +1,12 @@ +---+++ Vertex Edges + +[[CommonCLI][Common CLI Options]] + +Get the adjacent vertices or edges of the vertex with the specified direction. + +Usage: +$FALCON_HOME/bin/falcon metadata -edges -id <> -direction <> + +Example: +$FALCON_HOME/bin/falcon metadata -edges -id 4 -direction both +$FALCON_HOME/bin/falcon metadata -edges -id 4 -direction inE Added: falcon/trunk/general/src/site/twiki/falconcli/VertexMetadata.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/falconcli/VertexMetadata.twiki?rev=1730449&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/falconcli/VertexMetadata.twiki (added) +++ falcon/trunk/general/src/site/twiki/falconcli/VertexMetadata.twiki Mon Feb 15 05:48:00 2016 @@ -0,0 +1,11 @@ +---+++ Vertex + +[[CommonCLI][Common CLI Options]] + +Get the vertex with the specified id. + +Usage: +$FALCON_HOME/bin/falcon metadata -vertex -id <> + +Example: +$FALCON_HOME/bin/falcon metadata -vertex -id 4 Added: falcon/trunk/general/src/site/twiki/falconcli/VerticesMetadata.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/falconcli/VerticesMetadata.twiki?rev=1730449&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/falconcli/VerticesMetadata.twiki (added) +++ falcon/trunk/general/src/site/twiki/falconcli/VerticesMetadata.twiki Mon Feb 15 05:48:00 2016 @@ -0,0 +1,11 @@ +---+++ Vertices + +[[CommonCLI][Common CLI Options]] + +Get all vertices for a key index given the specified value. + +Usage: +$FALCON_HOME/bin/falcon metadata -vertices -key <> -value <> + +Example: +$FALCON_HOME/bin/falcon metadata -vertices -key type -value feed-instance Modified: falcon/trunk/general/src/site/twiki/index.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/index.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/index.twiki (original) +++ falcon/trunk/general/src/site/twiki/index.twiki Mon Feb 15 05:48:00 2016 @@ -28,7 +28,7 @@ to Falcon architecture and documentation describes steps to on-board a pipeline to Falcon. It also gives a sample pipeline for reference. [[EntitySpecification][Entity Specification]] gives complete details of all Falcon entities. -[[FalconCLI][Falcon CLI]] implements [[./restapi/ResourceList][Falcon's RESTful API]] and +[[falconcli/FalconCLI][Falcon CLI]] implements [[restapi/ResourceList][Falcon's RESTful API]] and describes various options for the command line utility provided by Falcon. Falcon provides OOTB [[HiveIntegration][lifecycle management for Tables in Hive (HCatalog)]] Modified: falcon/trunk/general/src/site/twiki/restapi/AdjacentVertices.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/AdjacentVertices.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/AdjacentVertices.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/AdjacentVertices.twiki Mon Feb 15 05:48:00 2016 @@ -9,6 +9,7 @@ Get a list of adjacent vertices or edges ---++ Parameters * :id is the id of the vertex. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. * :direction is the direction associated with the edges. To get the adjacent out vertices of vertex pass direction as out, in to get adjacent in vertices @@ -55,6 +56,26 @@ GET http://localhost:15000/api/metadata/ ---+++ Result +{ + "results":[ + { + "_id":"Q5V-4-5g", + "_type":"edge", + "_outV":4, + "_inV":8, + "_label":"output" + } + ], + "totalSize":1 +} + + +---+++ Rest Call + +GET http://localhost:15000/api/metadata/lineage/vertices/4/bothE?doAs=joe + +---+++ Result + { "results":[ { Modified: falcon/trunk/general/src/site/twiki/restapi/AdminStack.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/AdminStack.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/AdminStack.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/AdminStack.twiki Mon Feb 15 05:48:00 2016 @@ -8,14 +8,15 @@ Get stack trace of the falcon server. ---++ Parameters -None. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + ---++ Results Stack trace of the server. ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/admin/stack +GET http://localhost:15000/api/admin/stack?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/AdminVersion.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/AdminVersion.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/AdminVersion.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/AdminVersion.twiki Mon Feb 15 05:48:00 2016 @@ -8,14 +8,15 @@ Get version of the falcon server. ---++ Parameters -None. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + ---++ Results Version of the server. ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/admin/version +GET http://localhost:15000/api/admin/version?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/AllEdges.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/AllEdges.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/AllEdges.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/AllEdges.twiki Mon Feb 15 05:48:00 2016 @@ -8,7 +8,7 @@ Get all edges. ---++ Parameters -None. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results All edges in lineage graph. @@ -16,7 +16,7 @@ All edges in lineage graph. ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/lineage/edges/all +GET http://localhost:15000/api/metadata/lineage/edges/all?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/AllVertices.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/AllVertices.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/AllVertices.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/AllVertices.twiki Mon Feb 15 05:48:00 2016 @@ -8,7 +8,7 @@ Get all vertices. ---++ Parameters -None. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results All vertices in lineage graph. @@ -16,7 +16,7 @@ All vertices in lineage graph. ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/lineage/vertices/all +GET http://localhost:15000/api/metadata/lineage/vertices/all?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/Edge.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/Edge.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/Edge.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/Edge.twiki Mon Feb 15 05:48:00 2016 @@ -9,6 +9,7 @@ Gets the edge with specified id. ---++ Parameters * :id is the unique id of the edge. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Edge with the specified id. @@ -16,7 +17,7 @@ Edge with the specified id. ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/lineage/edges/Q6t-c-5g +GET http://localhost:15000/api/metadata/lineage/edges/Q6t-c-5g?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntityDefinition.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityDefinition.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityDefinition.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityDefinition.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Get definition of the entity. ---++ Parameters * :entity-type can be cluster, feed or process. * :entity-name is name of the entity. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Definition of the entity. @@ -17,7 +18,7 @@ Definition of the entity. ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/entities/definition/process/SampleProcess +GET http://localhost:15000/api/entities/definition/process/SampleProcess?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntityDelete.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityDelete.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityDelete.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityDelete.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Delete the specified entity. ---++ Parameters * :entity-type can be feed or process. * :entity-name is name of the feed or process. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Results of the delete operation. @@ -17,7 +18,7 @@ Results of the delete operation. ---++ Examples ---+++ Rest Call -DELETE http://localhost:15000/api/entities/delete/cluster/SampleProcess +DELETE http://localhost:15000/api/entities/delete/cluster/SampleProcess?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntityDependencies.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityDependencies.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityDependencies.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityDependencies.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Get dependencies of the entity. ---++ Parameters * :entity-type can be cluster, feed or process. * :entity-name is name of the entity. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Dependenciess of the entity. @@ -17,7 +18,7 @@ Dependenciess of the entity. ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/entities/dependencies/process/SampleProcess +GET http://localhost:15000/api/entities/dependencies/process/SampleProcess?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntityLineage.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityLineage.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityLineage.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityLineage.twiki Mon Feb 15 05:48:00 2016 @@ -9,6 +9,7 @@ It returns the graph depicting the relat ---++ Parameters * :pipeline is the name of the pipeline + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results It returns a json graph @@ -16,7 +17,7 @@ It returns a json graph ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/lineage/entities?pipeline=my-pipeline +GET http://localhost:15000/api/metadata/lineage/entities?pipeline=my-pipeline&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntityList.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityList.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityList.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityList.twiki Mon Feb 15 05:48:00 2016 @@ -8,13 +8,15 @@ Get list of the entities. ---++ Parameters - * :entity-type Valid options are cluster, feed or process. + * :entity-type Comma-separated entity types. Can be empty. Valid entity types are cluster, feed or process. * fields Fields of entity that the user wants to view, separated by commas. - * Valid options are STATUS, TAGS, PIPELINES. + * Valid options are STATUS, TAGS, PIPELINES, CLUSTERS. * nameseq Subsequence of entity name. Not case sensitive. * The entity name needs to contain all the characters in the subsequence in the same order. * Example 1: "sample1" will match the entity named "SampleFeed1-2". * Example 2: "mhs" will match the entity named "New-My-Hourly-Summary". + * tagkeys Keywords in tags, separated by comma. Not case sensitive. + * The returned entities will have tags that match all the tag keywords. * filterBy Filter results by list of field:value pairs. Example: filterBy=STATUS:RUNNING,PIPELINES:clickLogs * Supported filter fields are NAME, STATUS, PIPELINES, CLUSTER. * Query will do an AND among filterBy fields. @@ -25,9 +27,13 @@ Get list of the entities. * sortOrder Valid options are "asc" and "desc" * offset Show results from the offset, used for pagination. Defaults to 0. * numResults Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + * Note: + * We have two filtering parameters for entity tags: "tags" and "tagkeys". "tags" does the exact match in key=value fashion, while "tagkeys" finds all the entities with the given key as a substring in the tags. This "tagkeys" filter is introduced for the user who doesn't remember the exact tag but some keywords in the tag. It also helps users to save the time of typing long tags. + * The returned entities will match all the filtering criteria. ---++ Results -List of the entities. +Total number of results and a list of entities. ---++ Examples ---+++ Rest Call @@ -37,6 +43,7 @@ GET http://localhost:15000/api/entities/ ---+++ Result { + "totalResults":"2”, "entity": [ { "name": "SampleOutput", @@ -52,11 +59,41 @@ GET http://localhost:15000/api/entities/ ---+++ Rest Call +GET http://localhost:15000/api/entities/list + +---+++ Result + +{ + "totalResults":"4”, + "entity": [ + { + "name" : "SampleCluster1", + "type" : "cluster" + } + { + "name" : "SampleOutput", + "type" : "feed" + }, + { + "name" : "SampleInput", + "type" : "feed" + } + { + "name" : "SampleProcess1", + "type" : "process" + } + ] +} + + +---+++ Rest Call + GET http://localhost:15000/api/entities/list/feed?fields=status ---+++ Result { + "totalResults":"2”, "entity": [ { "name" : "SampleOutput", @@ -79,6 +116,7 @@ GET http://localhost:15000/api/entities/ ---+++ Result { + "totalResults":"10”, "entity": [ { "name" : "SampleProcess1", @@ -100,11 +138,12 @@ GET http://localhost:15000/api/entities/ ---+++ Rest Call -GET http://localhost:15000/api/entities/list/feed?nameseq=samplebill&numResults=2&offset=1&fields=status,clusters,tags +GET http://localhost:15000/api/entities/list/feed,process?nameseq=samplebill&tagkeys=billing,healthcare&numResults=2&offset=1&fields=status,clusters,tags&doAs=joe ---+++ Result { + "totalResults":"4”, "entity”:[ { "type":"FEED”, @@ -114,7 +153,7 @@ GET http://localhost:15000/api/entities/ "clusters": {"cluster":["SampleCluster1","primaryCluster”]} }, { - "type":"FEED”, + "type":"PROCESS”, "name":"SampleHealthBill”, "status":"SUBMITTED”, "tags”: {"tag":["related=healthcare","department=billingDepartment"]}, @@ -122,4 +161,4 @@ GET http://localhost:15000/api/entities/ } ] } - \ No newline at end of file + Modified: falcon/trunk/general/src/site/twiki/restapi/EntityResume.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityResume.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityResume.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityResume.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Resume a supended entity. ---++ Parameters * :entity-type can either be a feed or a process. * :entity-name is name of the entity. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Result of the resume command. @@ -17,7 +18,7 @@ Result of the resume command. ---++ Examples ---+++ Rest Call -POST http://localhost:15000/api/entities/resume/process/SampleProcess +POST http://localhost:15000/api/entities/resume/process/SampleProcess?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntitySchedule.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntitySchedule.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntitySchedule.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntitySchedule.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,9 @@ Schedule an entity. ---++ Parameters * :entity-type can either be a feed or a process. * :entity-name is name of the entity. + * skipDryRun : Optional query param, Falcon skips oozie dryrun when value is set to true. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + ---++ Results Result of the schedule command. @@ -17,7 +20,7 @@ Result of the schedule command. ---++ Examples ---+++ Rest Call -POST http://localhost:15000/api/entities/schedule/process/SampleProcess +POST http://localhost:15000/api/entities/schedule/process/SampleProcess?skipDryRun=false&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntityStatus.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityStatus.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityStatus.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityStatus.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Get status of the entity. ---++ Parameters * :entity-type can be cluster, feed or process. * :entity-name is name of the entity. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Status of the entity. @@ -17,7 +18,7 @@ Status of the entity. ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/entities/status/process/SampleProcess +GET http://localhost:15000/api/entities/status/process/SampleProcess?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntitySubmit.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntitySubmit.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntitySubmit.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntitySubmit.twiki Mon Feb 15 05:48:00 2016 @@ -8,7 +8,8 @@ Submit the given entity. ---++ Parameters -:entity-type can be cluster, feed or process. + * :entity-type can be cluster, feed or process. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Result of the submission. @@ -57,7 +58,7 @@ POST http://localhost:15000/api/entities ---+++ Rest Call -POST http://localhost:15000/api/entities/submit/process +POST http://localhost:15000/api/entities/submit/process?doAs=joe Modified: falcon/trunk/general/src/site/twiki/restapi/EntitySubmitAndSchedule.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntitySubmitAndSchedule.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntitySubmitAndSchedule.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntitySubmitAndSchedule.twiki Mon Feb 15 05:48:00 2016 @@ -9,6 +9,8 @@ Submits and schedules an entity. ---++ Parameters * :entity-type can either be a feed or a process. + * skipDryRun : Optional query param, Falcon skips oozie dryrun when value is set to true. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Result of the submit and schedule command. @@ -16,7 +18,7 @@ Result of the submit and schedule comman ---++ Examples ---+++ Rest Call -POST http://localhost:15000/api/entities/submitAndSchedule/process +POST http://localhost:15000/api/entities/submitAndSchedule/process?skipDryRun=false&doAs=joe Modified: falcon/trunk/general/src/site/twiki/restapi/EntitySummary.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntitySummary.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntitySummary.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntitySummary.twiki Mon Feb 15 05:48:00 2016 @@ -27,6 +27,7 @@ Given an EntityType and cluster, get lis * offset Show results from the offset, used for pagination. Defaults to 0. * numResults Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10. * numInstances Number of recent instances to show per entity. Only integers > 0 are valid, Default is 7. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Show entities along with summary of N instances for each entity. @@ -34,7 +35,7 @@ Show entities along with summary of N in ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/entities/summary/feed?cluster=primary-cluster&filterBy=STATUS:RUNNING&fields=status&tags=consumer=consumer@xyz.com&orderBy=name&offset=0&numResults=1&numInstances=2 +GET http://localhost:15000/api/entities/summary/feed?cluster=primary-cluster&filterBy=STATUS:RUNNING&fields=status&tags=consumer=consumer@xyz.com&orderBy=name&offset=0&numResults=1&numInstances=2&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntitySuspend.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntitySuspend.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntitySuspend.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntitySuspend.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Suspend an entity. ---++ Parameters * :entity-type can either be a feed or a process. * :entity-name is name of the entity. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Status of the entity. @@ -17,7 +18,7 @@ Status of the entity. ---++ Examples ---+++ Rest Call -POST http://localhost:15000/api/entities/suspend/process/SampleProcess +POST http://localhost:15000/api/entities/suspend/process/SampleProcess?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/EntityTouch.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityTouch.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityTouch.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityTouch.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,8 @@ Force updates the entity. ---++ Parameters * :entity-type can be feed or process. * :entity-name is name of the feed or process. + * skipDryRun : Optional query param, Falcon skips oozie dryrun when value is set to true. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Result of the validation. @@ -17,7 +19,7 @@ Result of the validation. ---++ Examples ---+++ Rest Call -POST http://localhost:15000/api/entities/touch/process/SampleProcess +POST http://localhost:15000/api/entities/touch/process/SampleProcess?skipDryRun=true&doAs=joe ---+++ Result @@ -26,4 +28,4 @@ POST http://localhost:15000/api/entities "message": "touch\/default\/SampleProcess updated successfully\n\n", "status": "SUCCEEDED" } - \ No newline at end of file + Modified: falcon/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,8 @@ Updates the submitted entity. ---++ Parameters * :entity-type can be feed or process. * :entity-name is name of the feed or process. + * skipDryRun : Optional query param, Falcon skips oozie dryrun when value is set to true. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Result of the validation. @@ -17,7 +19,7 @@ Result of the validation. ---++ Examples ---+++ Rest Call -POST http://localhost:15000/api/entities/update/process/SampleProcess +POST http://localhost:15000/api/entities/update/process/SampleProcess?skipDryRun=false&doAs=joe Modified: falcon/trunk/general/src/site/twiki/restapi/EntityValidate.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityValidate.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityValidate.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityValidate.twiki Mon Feb 15 05:48:00 2016 @@ -8,7 +8,9 @@ Validates the submitted entity. ---++ Parameters -:entity-type can be cluster, feed or process. + * :entity-type can be cluster, feed or process. + * skipDryRun : Optional query param, Falcon skips oozie dryrun when value is set to true. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Result of the validation. @@ -44,7 +46,7 @@ POST http://localhost:15000/api/entities ---+++ Rest Call -POST http://localhost:15000/api/entities/validate/feed +POST http://localhost:15000/api/entities/validate/feed?skipDryRun=true @@ -121,7 +123,7 @@ xmlns:xsi="http://www.w3.org/2001/XMLSch ---+++ Rest Call -POST http://localhost:15000/api/entities/validate/process +POST http://localhost:15000/api/entities/validate/process?skipDryRun=false&doAs=joe Modified: falcon/trunk/general/src/site/twiki/restapi/FeedInstanceListing.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/FeedInstanceListing.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/FeedInstanceListing.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/FeedInstanceListing.twiki Mon Feb 15 05:48:00 2016 @@ -14,6 +14,7 @@ Get falcon feed instance availability. * end Show instances up to this date. Date format is yyyy-MM-dd'T'HH:mm'Z'. * Default is set to now. * colo Colo on which the query should be run. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Feed instance availability status @@ -21,7 +22,7 @@ Feed instance availability status ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/instance/listing/feed/SampleFeed?colo=*&start=2012-04-03T07:00Z +GET http://localhost:15000/api/instance/listing/feed/SampleFeed?colo=*&start=2012-04-03T07:00Z&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/FeedLookup.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/FeedLookup.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/FeedLookup.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/FeedLookup.twiki Mon Feb 15 05:48:00 2016 @@ -9,6 +9,7 @@ ---++ Parameters * path path of the instance for which you want to determine the feed. e.g. /data/project1/2014/10/10/23/ Path has to be the complete path and can't be a part of it. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Returns the name of the feed along with the location type(meta/data/stats) and cluster on which the given path belongs to this feed. @@ -16,7 +17,7 @@ Returns the name of the feed along with ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/entities/lookup/feed?path=/data/project1/2014/10/10/23 +GET http://localhost:15000/api/entities/lookup/feed?path=/data/project1/2014/10/10/23&doAs=joe ---+++ Result { @@ -33,4 +34,4 @@ GET http://localhost:15000/api/entities/ "clusterName": "My-cluster2" } ] -} \ No newline at end of file +} Added: falcon/trunk/general/src/site/twiki/restapi/FeedSLA.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/FeedSLA.twiki?rev=1730449&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/FeedSLA.twiki (added) +++ falcon/trunk/general/src/site/twiki/restapi/FeedSLA.twiki Mon Feb 15 05:48:00 2016 @@ -0,0 +1,56 @@ +---++ GET /api/entities/sla-alert/:entity-type + * Description + * Parameters + * Results + * Examples + +---++ Description + +Since: 0.8 + +This command lists all the feed instances which have missed sla and are still not available. If a feed instance missed +sla but is now available, then it will not be reported in results. The purpose of this API is alerting and hence it + doesn't return feed instances which missed SLA but are available as they don't require any action. + +---++ Parameters + * :entity-type Only valid option is feed. + * entity-name parameter to restrict results for a particular feed using feed's name. + * start start of the time window for nominal instances, inclusive. + * end end of the time window for nominal instances to be considered, default is treated as current time. + * colo name of the colo + + +---++ Results +Pending feed instances which missed SLA. + +---++ Examples +---+++ Rest Call + +GET http://localhost:15000/api/entities/sla-alert/feed?colo=*&start=2012-04-03T07:00Z + +---+++ Result + +{ + "status":"SUCCEEDED", + "message":"default/Success!\n", + "requestId":"default/885720178@qtp-495452957-6 - f6e82e9b-d23f-466b-82df-4fb8293ce9cf\n", + "instances":[ + {"cluster":"local","entityName":"out","entityType":"FEED","instanceTime":"2015-09-26T17:33:00+05:30","tags":"Missed SLA High"}, + {"cluster":"local","entityName":"out","entityType":"FEED","instanceTime":"2015-09-26T17:29:00+05:30","tags":"Missed SLA High"}, + {"cluster":"local","entityName":"out","entityType":"FEED","instanceTime":"2015-09-26T17:35:00+05:30","tags":"Missed SLA Low"}, + {"cluster":"local","entityName":"out","entityType":"FEED","instanceTime":"2015-09-26T17:30:00+05:30","tags":"Missed SLA High"}, + {"cluster":"local","entityName":"out","entityType":"FEED","instanceTime":"2015-09-26T17:34:00+05:30","tags":"Missed SLA High"}, + {"cluster":"local","entityName":"out","entityType":"FEED","instanceTime":"2015-09-26T17:31:00+05:30","tags":"Missed SLA High"}, + {"cluster":"local","entityName":"out","entityType":"FEED","instanceTime":"2015-09-26T17:32:00+05:30","tags":"Missed SLA High"} + ] +} + + +In case there are no pending instances which have missed sla the response will be like below: + +{ + "status":"SUCCEEDED", + "message":"default/Success!\n", + "requestId":"default/979808239@qtp-1243851750-3 - 8c7396c0-efe2-43e9-9aea-7ae6afea5fd6\n" +} + Modified: falcon/trunk/general/src/site/twiki/restapi/Graph.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/Graph.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/Graph.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/Graph.twiki Mon Feb 15 05:48:00 2016 @@ -8,7 +8,7 @@ Dump the graph. ---++ Parameters -None. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Serialize graph to a file configured using *.falcon.graph.serialize.path in Custom startup.properties. @@ -16,7 +16,7 @@ Serialize graph to a file configured usi ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/lineage/serialize +GET http://localhost:15000/api/metadata/lineage/serialize?doAs=joe ---+++ Result None. Added: falcon/trunk/general/src/site/twiki/restapi/InstanceDependencies.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceDependencies.twiki?rev=1730449&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceDependencies.twiki (added) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceDependencies.twiki Mon Feb 15 05:48:00 2016 @@ -0,0 +1,49 @@ +---++ GET /api/instance/dependencies/:entity-type/:entity-name + * Description + * Parameters + * Results + * Examples + +---++ Description +Get dependent instances for a particular instance. + +---++ Parameters + * :entity-type Valid options are feed or process. + * :entity-name Name of the entity + * instanceTime time of the given instance + * colo name of the colo + + +---++ Results +Dependent instances for the specified instance + +---++ Examples +---+++ Rest Call + +GET http://localhost:15000/api/instance/dependencies/feed/myFeed?colo=*&instanceTime=2012-04-03T07:00Z + +---+++ Result + +{ + 'status': 'SUCCEEDED', + 'message': 'default/Success!\n', + 'dependencies': [ + { + 'cluster': 'local', + 'entityName': 'consumer-process', + 'entityType': 'PROCESS', + 'instanceTime': '2014-12-18T00:00Z', + 'tags': 'Input' + }, + { + 'cluster': 'local', + 'entityName': 'producer-process', + 'entityType': 'PROCESS', + 'instanceTime': '2014-12-18T00:00Z', + 'tags': 'Output' + } + ], + 'requestId': 'default/1405883107@qtp-1501726962-6-0c2e690f-546b-47b0-a5ee-0365d4522a31\n' +} + + Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceKill.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceKill.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceKill.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceKill.twiki Mon Feb 15 05:48:00 2016 @@ -13,6 +13,7 @@ Kill currently running instance(s) of an * start is the start time of the instance(s) that you want to refer to * end is the end time of the instance(s) that you want to refer to * lifecycle can be Eviction/Replication(default) for feed and Execution(default) for process. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Result of the kill operation. @@ -20,7 +21,7 @@ Result of the kill operation. ---++ Examples ---+++ Rest Call -POST http://localhost:15000/api/instance/kill/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z +POST http://localhost:15000/api/instance/kill/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceList.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceList.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceList.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceList.twiki Mon Feb 15 05:48:00 2016 @@ -24,10 +24,21 @@ Get list of all instances of a given ent * sortOrder Valid options are "asc" and "desc" * offset Show results from the offset, used for pagination. Defaults to 0. * numResults Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results List of instances of given entity. +The possible instance status returned and its meaning are as follows: + * WAITING - The instance is waiting for the corresponding data(feed) instances to become available. + * READY - The instance is ready to be scheduled. But, is waiting for scheduling conditions to be met. For example, limitation on number of instances that can be run in parallel. + * RUNNING - The instance is running on the workflow engine. + * FAILED - The instance has failed during execution. + * KILLED - The instance has been killed either manually or by the system. + * SUCCEEDED - The instance has executed successfully. + * SKIPPED - This instance was not executed, but was skipped. For example, when the execution order is LAST_ONLY, the older instances are skipped. + * ERROR - There was error while executing this instance on the workflow engine. + * UNDEFINED - The status of the instance could not be determined. ---++ Examples ---+++ Rest Call @@ -56,7 +67,7 @@ GET http://localhost:15000/api/instance/ ---+++ Rest Call -GET http://localhost:15000/api/instance/list/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=STATUS:SUCCEEDED,CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2 +GET http://localhost:15000/api/instance/list/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=STATUS:SUCCEEDED,CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceLogs.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceLogs.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceLogs.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceLogs.twiki Mon Feb 15 05:48:00 2016 @@ -25,6 +25,7 @@ Get log of a specific instance of an ent * sortOrder Valid options are "asc" and "desc" * offset Show results from the offset, used for pagination. Defaults to 0. * numResults Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Log of specified instance. @@ -63,7 +64,7 @@ GET http://localhost:15000/api/instance/ ---+++ Rest Call -GET http://localhost:15000/api/instance/logs/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=STATUS:SUCCEEDED,CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2 +GET http://localhost:15000/api/instance/logs/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=STATUS:SUCCEEDED,CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceParams.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceParams.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceParams.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceParams.twiki Mon Feb 15 05:48:00 2016 @@ -13,6 +13,7 @@ Get the params passed to the workflow fo * start should be the nominal time of the instance for which you want the params to be returned * colo Colo on which the query should be run. * lifecycle Valid lifecycles for feed are Eviction/Replication(default) and for process is Execution(default). + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results @@ -21,7 +22,7 @@ List of instances currently running. ---++ Examples ---+++ Rest Call -http://userqa.user.com:16000/api/instance/params/process/Sample-Process?start=2014-10-01T11:00Z&colo=* +http://userqa.user.com:16000/api/instance/params/process/Sample-Process?start=2014-10-01T11:00Z&colo=*&doAs=joe ---+++ Result { @@ -79,4 +80,4 @@ http://userqa.user.com:16000/api/instanc } } ] -} \ No newline at end of file +} Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceRerun.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceRerun.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceRerun.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceRerun.twiki Mon Feb 15 05:48:00 2016 @@ -14,6 +14,7 @@ Rerun instances of an entity. On issuing * end is the end time of the instance that you want to refer to * lifecycle can be Eviction/Replication(default) for feed and Execution(default) for process. * force can be used to forcefully rerun the entire instance. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Results of the rerun command. @@ -43,7 +44,7 @@ POST http://localhost:15000/api/instance -POST http://localhost:15000/api/instance/rerun/process/SampleProcess?colo=*&start=2013-04-03T07:00Z&end=2014-04-03T07:00Z&force=true +POST http://localhost:15000/api/instance/rerun/process/SampleProcess?colo=*&start=2013-04-03T07:00Z&end=2014-04-03T07:00Z&force=true&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceResume.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceResume.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceResume.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceResume.twiki Mon Feb 15 05:48:00 2016 @@ -13,29 +13,31 @@ Resume suspended instances of an entity. * start is the start time of the instance(s) that you want to refer to * end is the end time of the instance(s) that you want to refer to * lifecycle can be Eviction/Replication(default) for feed and Execution(default) for process. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + ---++ Results Results of the resume command. ---++ Examples ---+++ Rest Call - -POST http://localhost:15000/api/instance/resume/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z - ----+++ Result - -{ - "instances": [ - { - "details": "", - "startTime": "2013-10-21T15:19:57-07:00", - "cluster": "primary-cluster", - "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W", - "status": "RUNNING", - "instance": "2012-04-03T07:00Z" - } - ], - "requestId": "default\/e88ff2e0-2af7-4829-a360-f92e95be2981\n", - "message": "default\/RESUME\n", - "status": "SUCCEEDED" -} - + + POST http://localhost:15000/api/instance/resume/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z&doAs=joe + + ---+++ Result + + { + "instances": [ + { + "details": "", + "startTime": "2013-10-21T15:19:57-07:00", + "cluster": "primary-cluster", + "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W", + "status": "RUNNING", + "instance": "2012-04-03T07:00Z" + } + ], + "requestId": "default\/e88ff2e0-2af7-4829-a360-f92e95be2981\n", + "message": "default\/RESUME\n", + "status": "SUCCEEDED" + } + Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceRunning.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceRunning.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceRunning.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceRunning.twiki Mon Feb 15 05:48:00 2016 @@ -20,7 +20,7 @@ Get a list of instances currently runnin * sortOrder Valid options are "asc" and "desc" * offset Show results from the offset, used for pagination. Defaults to 0. * numResults Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10. - + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results List of instances currently running. @@ -50,7 +50,7 @@ GET http://localhost:15000/api/instance/ ---+++ Rest Call -GET http://localhost:15000/api/instance/running/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2 +GET http://localhost:15000/api/instance/running/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2&doAs=joe ---+++ Result @@ -82,4 +82,3 @@ GET http://localhost:15000/api/instance/ } - Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceStatus.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceStatus.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceStatus.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceStatus.twiki Mon Feb 15 05:48:00 2016 @@ -24,7 +24,7 @@ Get status of a specific instance of an * sortOrder Valid options are "asc" and "desc" * offset Show results from the offset, used for pagination. Defaults to 0. * numResults Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10. - + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Status of the specified instance along with job urls for all actions of user workflow and non-succeeded actions of the main-workflow. @@ -64,7 +64,7 @@ GET https://localhost:15443/api/instance ---+++ Rest Call -GET http://localhost:15000/api/instance/status/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=STATUS:SUCCEEDED,CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2 +GET http://localhost:15000/api/instance/status/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=STATUS:SUCCEEDED,CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceSummary.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceSummary.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceSummary.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceSummary.twiki Mon Feb 15 05:48:00 2016 @@ -8,14 +8,24 @@ Get summary of instance/instances of an entity. ---++ Parameters - * :entity-type Valid options are cluster, feed or process. + * :entity-type Valid options are feed or process. * :entity-name Name of the entity. * start Show instances from this date. Date format is yyyy-MM-dd'T'HH:mm'Z'. - * By default, it is set to (end - (10 * entityFrequency)). + * By default, it is set to (end - (10 * entityFrequency)). * end Show instances up to this date. Date format is yyyy-MM-dd'T'HH:mm'Z'. - * Default is set to now. + * Default is set to now. * colo Colo on which the query should be run. * lifecycle Valid lifecycles for feed are Eviction/Replication(default) and for process is Execution(default). + * filterBy Filter results by list of field:value pairs. + Example1: filterBy=STATUS:RUNNING,CLUSTER:primary-cluster + Example2: filterBy=Status:RUNNING,Status:KILLED + * Supported filter fields are STATUS, CLUSTER + * Query will do an AND among filterBy fields. + * orderBy Field by which results should be ordered. + * Supports ordering by "cluster". + * sortOrder Valid options are "asc" and "desc" + Example: orderBy=cluster sortOrder=asc + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Summary of the instances over the specified time range @@ -39,10 +49,66 @@ GET http://localhost:15000/api/instance/ "entry": { "key":"SUCCEEDED", - "key2":"value" + "value":"value" } } } } } + +---+++ Rest Call + +GET https://localhost:16443/api/instance/summary/process/WordCount?filterBy=Status:KILLED,Status:RUNNING&start=2015-06-24T16:00Z&end=2015-06-24T23:00Z&colo=* + +---+++ Result + +{ + "status":"SUCCEEDED", + "message":"local/SUMMARY\n", + "requestId":"local/1246061948@qtp-1059149611-5 - 34d8c3bb-f461-4fd5-87cd-402c9c6b1ed2\n", + "instancesSummary":[ + { + "cluster":"local", + "map":{ + "entry":{ + "key":"RUNNING", + "value":"1" + }, + "entry":{ + "key":"KILLED", + "value":"1" + } + } + } + ] +} + + +---+++ Rest Call + +GET https://localhost:16443/api/instance/summary/process/WordCount?orderBy=cluster&sortOrder=asc&start=2015-06-24T16:00Z&end=2015-06-24T23:00Z&colo=*&doAs=joe + +---+++ Result + +{ + "status":"SUCCEEDED", + "message":"local/SUMMARY\n", + "requestId":"local/1246061948@qtp-1059149611-5 - 42e2040d-6b6e-4bfd-a090-83db5ed1a429\n", + "instancesSummary":[ + { + "cluster":"local", + "map":{ + "entry":{ + "key":"SUCCEEDED", + "value":"6" + }, + "entry":{ + "key":"KILLED", + "value":"1" + } + } + } + ] +} + Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceSuspend.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceSuspend.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceSuspend.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceSuspend.twiki Mon Feb 15 05:48:00 2016 @@ -13,6 +13,7 @@ Suspend instances of an entity. * start is the start time of the instance(s) that you want to refer to * end is the end time of the instance(s) that you want to refer to * lifecycle can be Eviction/Replication(default) for feed and Execution(default) for process. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Results of the suspend command. @@ -20,7 +21,7 @@ Results of the suspend command. ---++ Examples ---+++ Rest Call -POST http://localhost:15000/api/instance/suspend/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z +POST http://localhost:15000/api/instance/suspend/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z&doAs=joe ---+++ Result @@ -40,4 +41,4 @@ POST http://localhost:15000/api/instance "message": "default\/SUSPEND\n", "status": "SUCCEEDED" } - + \ No newline at end of file Modified: falcon/trunk/general/src/site/twiki/restapi/MetadataList.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/MetadataList.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/MetadataList.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/MetadataList.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Get all dimensions of specified type. ---++ Parameters * :type Valid dimension types are cluster_entity,feed_entity, process_entity, user, colo, tags, groups, pipelines * cluster Show dimensions related to this cluster. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results @@ -18,7 +19,7 @@ List of dimensions that match requested ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/discovery/process_entity/list?cluster=primary-cluster +GET http://localhost:15000/api/metadata/discovery/process_entity/list?cluster=primary-cluster&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/MetadataRelations.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/MetadataRelations.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/MetadataRelations.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/MetadataRelations.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Get all relations of a specific dimensio ---++ Parameters * :type Valid dimension types are cluster_entity,feed_entity, process_entity, user, colo, tags, groups, pipelines * :name Name of the dimension. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Get all relations of a specific dimension. @@ -17,7 +18,7 @@ Get all relations of a specific dimensio ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/discovery/process_entity/sample-process/relations +GET http://localhost:15000/api/metadata/discovery/process_entity/sample-process/relations?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/ResourceList.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/ResourceList.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/ResourceList.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/ResourceList.twiki Mon Feb 15 05:48:00 2016 @@ -29,6 +29,8 @@ curl -i --negotiate -u : "http://: See also: [[../Security.twiki][Security in Falcon]] +The current version of the rest api's documentation is also hosted on the Falcon server and Prism Server (in distributed mode) at the url http://:/docs + ---++ REST Call on Admin Resource | *Call Type* | *Resource* | *Description* | @@ -52,6 +54,7 @@ See also: [[../Security.twiki][Security | GET | [[EntityList][api/entities/list/:entity-type]] | Get the list of entities | | GET | [[EntitySummary][api/entities/summary/:entity-type/:cluster]] | Get instance summary of all entities | | GET | [[EntityDependencies][api/entities/dependencies/:entity-type/:entity-name]] | Get the dependencies of the entity | +| GET | [[FeedSLA][api/entities/sla-alert/:entity-type]] | Get pending feed instances which missed sla | | GET | [[FeedLookup][api/entities/lookup/feed/]] | Get feed for given path | ---++ REST Call on Feed and Process Instances @@ -66,7 +69,9 @@ See also: [[../Security.twiki][Security | POST | [[InstanceResume][api/instance/resume/:entity-type/:entity-name]] | Resume a given instance | | POST | [[InstanceRerun][api/instance/rerun/:entity-type/:entity-name]] | Rerun a given instance | | GET | [[InstanceLogs][api/instance/logs/:entity-type/:entity-name]] | Get logs of a given instance | +| GET | [[Triage][api/instance/triage/:entity-type/:entity-name]] | Triage an instance to see it's stuck lineage | | GET | [[InstanceSummary][api/instance/summary/:entity-type/:entity-name]] | Return summary of instances for an entity | +| GET | [[InstanceDependency][api/instance/dependencies/:entity-type/:entity-name]] | Return dependent instances for a given instance | ---++ REST Call on Metadata Lineage Resource Added: falcon/trunk/general/src/site/twiki/restapi/Triage.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/Triage.twiki?rev=1730449&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/Triage.twiki (added) +++ falcon/trunk/general/src/site/twiki/restapi/Triage.twiki Mon Feb 15 05:48:00 2016 @@ -0,0 +1,45 @@ +---++ GET api/instance/triage/:entity-type/:entity-name + * Description + * Parameters + * Results + * Examples + +---++ Description +Given a feed/process instance this command traces it's ancestors to find what all ancestors have failed. It's useful if +lot of instances are failing in a pipeline as it then finds out the root cause of the pipeline being stuck. + + +---++ Parameters + * :entity-type type of entity(feed/process). + * :entity-name name of the feed/process. + * :start instance time of the entity instance. + * :colo name of the colo on which you want to triage + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +It returns a json graph + +---++ Examples +---+++ Rest Call + +GET http://localhost:15000/api/instance/triage/feed/my-feed?start=2015-03-02T00:00Z&colo=local&doAs=joe + +---+++ Result + +{ + "vertices": ["(FEED) my-feed (2015-03-02T00:00Z) [Unavailable]", "(PROCESS) producer-process (2015-03-01T10:00Z) [TIMEDOUT]", "(FEED) input-feed-for-producer (2015-03-01T00:00Z) [Available]"], + "edges": + [ + { + "from" : "(PROCESS) producer-process (2015-03-01T10:00Z) [TIMEDOUT]", + "to" : "(FEED) my-feed (2015-03-02T00:00Z) [Unavailable]", + "label" : "produces" + }, + { + "from" : "(FEED) input-feed-for-producer (2015-03-01T00:00Z) [Available]", + "to" : "(PROCESS) producer-process (2015-03-01T10:00Z) [TIMEDOUT]", + "label" : "consumed by" + } + ] +} + Modified: falcon/trunk/general/src/site/twiki/restapi/Vertex.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/Vertex.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/Vertex.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/Vertex.twiki Mon Feb 15 05:48:00 2016 @@ -9,6 +9,7 @@ Gets the vertex with specified id. ---++ Parameters * :id is the unique id of the vertex. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Vertex with the specified id. @@ -16,7 +17,7 @@ Vertex with the specified id. ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/lineage/vertices/4 +GET http://localhost:15000/api/metadata/lineage/vertices/4?doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/VertexProperties.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/VertexProperties.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/VertexProperties.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/VertexProperties.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Gets the properties of the vertex with s ---++ Parameters * :id is the unique id of the vertex. * :relationships has default value of false. Pass true if relationships should be fetched. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results Properties associated with the specified vertex. @@ -17,7 +18,7 @@ Gets the properties of the vertex with s ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/lineage/vertices/properties/40004?relationships=true +GET http://localhost:15000/api/metadata/lineage/vertices/properties/40004?relationships=true&doAs=joe ---+++ Result Modified: falcon/trunk/general/src/site/twiki/restapi/Vertices.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/Vertices.twiki?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/Vertices.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/Vertices.twiki Mon Feb 15 05:48:00 2016 @@ -10,6 +10,7 @@ Get all vertices for a key index given t ---++ Parameters * :key is the key to be matched. * :value is the associated value of the key. + * doAs allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. ---++ Results All vertices matching given property key and a value. @@ -17,7 +18,7 @@ All vertices matching given property key ---++ Examples ---+++ Rest Call -GET http://localhost:15000/api/metadata/lineage/vertices?key=name&value=sampleIngestProcess +GET http://localhost:15000/api/metadata/lineage/vertices?key=name&value=sampleIngestProcess&doAs=joe ---+++ Result @@ -34,4 +35,4 @@ GET http://localhost:15000/api/metadata/ ], "totalSize": 1 } - + \ No newline at end of file Modified: falcon/trunk/pom.xml URL: http://svn.apache.org/viewvc/falcon/trunk/pom.xml?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/pom.xml (original) +++ falcon/trunk/pom.xml Mon Feb 15 05:48:00 2016 @@ -21,7 +21,7 @@ 4.0.0 org.apache.falcon falcon-website - 0.8-SNAPSHOT + 0.9-SNAPSHOT pom Apache Falcon @@ -286,6 +286,26 @@ committer + Hortonworks + + + pavankumar + Pavan Kumar Kolamuri + pavankumar@apache.org + +05:30 + + committer + + Inmobi + + + peeyushb + Peeyush Bishnoi + peeyushb@apache.org + +05:30 + + committer + Hortonworks Modified: falcon/trunk/releases/pom.xml URL: http://svn.apache.org/viewvc/falcon/trunk/releases/pom.xml?rev=1730449&r1=1730448&r2=1730449&view=diff ============================================================================== --- falcon/trunk/releases/pom.xml (original) +++ falcon/trunk/releases/pom.xml Mon Feb 15 05:48:00 2016 @@ -22,7 +22,7 @@ org.apache.falcon falcon-website - 0.8-SNAPSHOT + 0.10-SNAPSHOT falcon-website-releases 0.2 @@ -38,6 +38,7 @@ 0.6.1 0.7 0.8 + 0.9