Return-Path: X-Original-To: apmail-falcon-commits-archive@minotaur.apache.org Delivered-To: apmail-falcon-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6CF5E11D15 for ; Thu, 21 Aug 2014 17:31:13 +0000 (UTC) Received: (qmail 44205 invoked by uid 500); 21 Aug 2014 17:31:12 -0000 Delivered-To: apmail-falcon-commits-archive@falcon.apache.org Received: (qmail 43573 invoked by uid 500); 21 Aug 2014 17:31:12 -0000 Mailing-List: contact commits-help@falcon.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.incubator.apache.org Delivered-To: mailing list commits@falcon.incubator.apache.org Received: (qmail 43561 invoked by uid 99); 21 Aug 2014 17:31:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Aug 2014 17:31:12 +0000 X-ASF-Spam-Status: No, hits=-2000.7 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 21 Aug 2014 17:30:46 +0000 Received: (qmail 39026 invoked by uid 99); 21 Aug 2014 17:30:44 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Aug 2014 17:30:44 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id D19AD9C0945; Thu, 21 Aug 2014 17:30:43 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: arpit@apache.org To: commits@falcon.incubator.apache.org Date: Thu, 21 Aug 2014 17:30:47 -0000 Message-Id: In-Reply-To: <80dc65e0ddcc42b5bcf692abce60d0ff@git.apache.org> References: <80dc65e0ddcc42b5bcf692abce60d0ff@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [05/18] git commit: FALCON-614 Add pipeline element to process entity. Contributed by Balu Vellanki X-Virus-Checked: Checked by ClamAV on apache.org FALCON-614 Add pipeline element to process entity. Contributed by Balu Vellanki Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/69e0bff3 Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/69e0bff3 Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/69e0bff3 Branch: refs/heads/FALCON-585 Commit: 69e0bff317172114e2f86d9b8808c55ec850abaa Parents: 3fc1e8d Author: Venkatesh Seetharam Authored: Mon Aug 18 16:10:38 2014 -0700 Committer: Venkatesh Seetharam Committed: Mon Aug 18 16:10:38 2014 -0700 ---------------------------------------------------------------------- CHANGES.txt | 3 ++ client/src/main/resources/process-0.1.xsd | 14 +++++ .../entity/parser/ProcessEntityParserTest.java | 20 +++++++ .../resources/config/process/process-0.1.xml | 2 + .../config/process/process-bad-pipeline.xml | 56 ++++++++++++++++++++ docs/src/site/twiki/EntitySpecification.twiki | 17 ++++++ 6 files changed, 112 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/69e0bff3/CHANGES.txt ---------------------------------------------------------------------- diff --git a/CHANGES.txt b/CHANGES.txt index 085773b..b4dfdc4 100755 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -5,6 +5,9 @@ Trunk (Unreleased) INCOMPATIBLE CHANGES NEW FEATURES + FALCON-614 Add pipeline element to process entity + (Balu Vellanki via Venkatesh Seetharam) + FALCON-588 Baselining designer code. (samar via Shwetha GS) FALCON-400 Add Authorization for Entities (Venkatesh Seetharam) http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/69e0bff3/client/src/main/resources/process-0.1.xsd ---------------------------------------------------------------------- diff --git a/client/src/main/resources/process-0.1.xsd b/client/src/main/resources/process-0.1.xsd index cd4f5d2..06a2fe4 100644 --- a/client/src/main/resources/process-0.1.xsd +++ b/client/src/main/resources/process-0.1.xsd @@ -64,6 +64,15 @@ + + + + pipelines: a process specifies an optional list of comma separated pipelines, + separated by comma, which is used for classification of processes. + Example: dataReplicationPipeline, clickStreamPipeline + + + Defines the clusters where the workflow should run @@ -353,6 +362,11 @@ + + + + + http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/69e0bff3/common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java ---------------------------------------------------------------------- diff --git a/common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java b/common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java index 3513dab..80a9cc7 100644 --- a/common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java +++ b/common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java @@ -95,6 +95,10 @@ public class ProcessEntityParserTest extends AbstractTestBase { Assert.assertEquals(process.getFrequency().toString(), "hours(1)"); Assert.assertEquals(process.getEntityType(), EntityType.PROCESS); + Assert.assertEquals(process.getTags(), + "consumer=consumer@xyz.com, owner=producer@xyz.com, department=forecasting"); + Assert.assertEquals(process.getPipelines(), "testPipeline,dataReplication_Pipeline"); + Assert.assertEquals(process.getInputs().getInputs().get(0).getName(), "impression"); Assert.assertEquals(process.getInputs().getInputs().get(0).getFeed(), "impressionFeed"); Assert.assertEquals(process.getInputs().getInputs().get(0).getStart(), "today(0,0)"); @@ -463,4 +467,20 @@ public class ProcessEntityParserTest extends AbstractTestBase { StartupProperties.get().setProperty("falcon.security.authorization.enabled", "false"); } } + + /** + * A negative test for validating pipelines tag which is comma separated values. + * @throws FalconException + */ + @Test + public void testPipelineTags() throws FalconException { + try { + InputStream stream = this.getClass().getResourceAsStream("/config/process/process-bad-pipeline.xml"); + + parser.parse(stream); + Assert.fail("org.xml.sax.SAXParseException should have been thrown."); + } catch (FalconException e) { + Assert.assertEquals(javax.xml.bind.UnmarshalException.class, e.getCause().getClass()); + } + } } http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/69e0bff3/common/src/test/resources/config/process/process-0.1.xml ---------------------------------------------------------------------- diff --git a/common/src/test/resources/config/process/process-0.1.xml b/common/src/test/resources/config/process/process-0.1.xml index 6e27577..99a0376 100644 --- a/common/src/test/resources/config/process/process-0.1.xml +++ b/common/src/test/resources/config/process/process-0.1.xml @@ -17,6 +17,8 @@ limitations under the License. --> + consumer=consumer@xyz.com, owner=producer@xyz.com, department=forecasting + testPipeline,dataReplication_Pipeline http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/69e0bff3/common/src/test/resources/config/process/process-bad-pipeline.xml ---------------------------------------------------------------------- diff --git a/common/src/test/resources/config/process/process-bad-pipeline.xml b/common/src/test/resources/config/process/process-bad-pipeline.xml new file mode 100644 index 0000000..e506bd9 --- /dev/null +++ b/common/src/test/resources/config/process/process-bad-pipeline.xml @@ -0,0 +1,56 @@ + + + + consumer=consumer@xyz.com, owner=producer@xyz.com, department=forecasting + etl-pipeline fetl-pipeline, , + + + + + + 1 + LIFO + hours(1) + + + + + + + + + + + + + + + + + + + + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/69e0bff3/docs/src/site/twiki/EntitySpecification.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/EntitySpecification.twiki b/docs/src/site/twiki/EntitySpecification.twiki index 4eb4b17..df572cb 100644 --- a/docs/src/site/twiki/EntitySpecification.twiki +++ b/docs/src/site/twiki/EntitySpecification.twiki @@ -280,6 +280,23 @@ Syntax: +---+++ Tags +An optional list of comma separated tags which are used for classification of processes. +Syntax: + +... + consumer=consumer@xyz.com, owner=producer@xyz.com, department=forecasting + + +---+++ Pipelines +An optional list of comma separated word strings, specifies the data processing pipeline(s) to which this process belongs. +Only letters, numbers and underscore are allowed for pipeline string. +Syntax: + +... + test_Pipeline, dataReplication, clickStream_pipeline + + ---+++ Cluster The cluster on which the workflow should run. A process should contain one or more clusters. Cluster definition for the cluster name gives the end points for workflow execution, name node, job tracker, messaging and so on. Each cluster inturn has validity mentioned, which tell the times between which the job should run on that specified cluster. Syntax: