falcon-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From suh...@apache.org
Subject incubator-falcon git commit: FALCON-721. Add SLA for Feeds. Contributed by Ajay Yadav
Date Fri, 07 Nov 2014 09:29:14 GMT
Repository: incubator-falcon
Updated Branches:
  refs/heads/master 6113b70b0 -> dbdb70751


FALCON-721. Add SLA for Feeds. Contributed by Ajay Yadav


Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/dbdb7075
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/dbdb7075
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/dbdb7075

Branch: refs/heads/master
Commit: dbdb70751cbba9992ecf7fcda200bd5ba2734bed
Parents: 6113b70
Author: Suhas V <suhas.v@inmobi.com>
Authored: Fri Nov 7 14:58:59 2014 +0530
Committer: Suhas V <suhas.v@inmobi.com>
Committed: Fri Nov 7 14:58:59 2014 +0530

----------------------------------------------------------------------
 CHANGES.txt                                     |  3 ++
 client/src/main/resources/feed-0.1.xsd          | 19 ++++++++++
 .../org/apache/falcon/entity/FeedHelper.java    | 11 ++++++
 .../falcon/entity/parser/FeedEntityParser.java  | 37 ++++++++++++++++++++
 .../entity/parser/FeedEntityParserTest.java     | 26 ++++++++++++++
 .../src/test/resources/config/feed/feed-0.1.xml |  2 ++
 docs/src/site/twiki/EntitySpecification.twiki   | 20 +++++++++--
 7 files changed, 115 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/dbdb7075/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 1321734..b5e569b 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -37,7 +37,10 @@ Trunk (Unreleased)
    FALCON-263 API to get workflow parameters. (pavan kumar kolamuri via Shwetha GS)
 
   IMPROVEMENTS
+   FALCON-721 Add SLA for Feeds (Ajay Yadav via Suhas Vasu)
+
    FALCON-813 Expose job id for running jobs in Falcon (Suhas Vasu)
+
    FALCON-834 Propagate request id in the response to help trace and debug
    failures in merlin (Venkatesh Seetharam)
 

http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/dbdb7075/client/src/main/resources/feed-0.1.xsd
----------------------------------------------------------------------
diff --git a/client/src/main/resources/feed-0.1.xsd b/client/src/main/resources/feed-0.1.xsd
index 3c96d80..0168a35 100644
--- a/client/src/main/resources/feed-0.1.xsd
+++ b/client/src/main/resources/feed-0.1.xsd
@@ -75,6 +75,10 @@
                 frequency and a periodicity which specifies the frequency by which
                 this feed is generated. ex: it can be generated every hour, every 5 minutes,
daily, weekly etc.
                 valid frequency type for a feed are minutes, hours, days, months.
+                sla: A feed can have SLA and each SLA has two properties - slaLow and slaHigh.
Both slaLow and slaHigh
+                are written using expressions like frequency. slaLow is intended to serve
for alerting for feeds which
+                are in danger of missing their availability SLAs. slaHigh is intended to
serve for reporting the feeds
+                which missed their SLAs. SLAs are relative to feed instance time.
             </xs:documentation>
         </xs:annotation>
         <xs:sequence>
@@ -92,6 +96,7 @@
             <xs:element type="group-type" name="groups" minOccurs="0"/>
             <xs:element type="xs:string" name="availabilityFlag" minOccurs="0"/>
             <xs:element type="frequency-type" name="frequency"/>
+            <xs:element type="sla" name="sla" minOccurs="0"/>
             <xs:element name="timezone" minOccurs="0" default="UTC">
                 <xs:simpleType>
                     <xs:annotation>
@@ -142,6 +147,7 @@
         <xs:sequence>
             <xs:element type="validity" name="validity"/>
             <xs:element type="retention" name="retention"/>
+            <xs:element type="sla" name="sla" minOccurs="0" maxOccurs="1"/>
             <xs:choice minOccurs="0" maxOccurs="1">
                 <xs:element type="locations" name="locations" minOccurs="0"/>
                 <xs:element type="catalog-table" name="table"/>
@@ -151,6 +157,7 @@
         <xs:attribute type="cluster-type" name="type" use="optional"/>
         <xs:attribute type="xs:string" name="partition" use="optional"/>
         <xs:attribute type="frequency-type" name="delay" use="optional" /> 
+
     </xs:complexType>
     <xs:complexType name="partitions">
         <xs:annotation>
@@ -197,6 +204,18 @@
         <xs:attribute type="date-time-type" name="start" use="required"/>
         <xs:attribute type="date-time-type" name="end" use="required"/>
     </xs:complexType>
+    <xs:complexType name="sla">
+        <xs:annotation>
+            <xs:documentation>
+                sla has two properties - slaLow and slaHigh. Both slaLow and slaHigh
+                are written using expressions like frequency. slaLow is intended to serve
for alerting for feeds which
+                are in danger of missing their availability SLAs. slaHigh is intended to
serve for reporting the feeds
+                which missed their SLAs. SLAs are relative to feed instance time.
+            </xs:documentation>
+        </xs:annotation>
+        <xs:attribute type="frequency-type" name="slaLow" use="required"/>
+        <xs:attribute type="frequency-type" name="slaHigh" use="required"/>
+    </xs:complexType>
     <xs:complexType name="locations">
         <xs:annotation>
             <xs:documentation>

http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/dbdb7075/common/src/main/java/org/apache/falcon/entity/FeedHelper.java
----------------------------------------------------------------------
diff --git a/common/src/main/java/org/apache/falcon/entity/FeedHelper.java b/common/src/main/java/org/apache/falcon/entity/FeedHelper.java
index 4532669..73b55d3 100644
--- a/common/src/main/java/org/apache/falcon/entity/FeedHelper.java
+++ b/common/src/main/java/org/apache/falcon/entity/FeedHelper.java
@@ -32,6 +32,7 @@ import org.apache.falcon.entity.v0.feed.Feed;
 import org.apache.falcon.entity.v0.feed.Location;
 import org.apache.falcon.entity.v0.feed.LocationType;
 import org.apache.falcon.entity.v0.feed.Locations;
+import org.apache.falcon.entity.v0.feed.Sla;
 import org.apache.falcon.expression.ExpressionHelper;
 import org.apache.falcon.resource.APIResult;
 import org.apache.falcon.resource.FeedInstanceResult;
@@ -222,6 +223,16 @@ public final class FeedHelper {
         return feedLocations == null ? null : feedLocations.getLocations();
     }
 
+    public static Sla getSLAs(Cluster cluster, Feed feed) {
+        final Sla clusterSla = cluster.getSla();
+        if (clusterSla != null) {
+            return clusterSla;
+        }
+
+        final Sla feedSla = feed.getSla();
+        return feedSla == null ? null : feedSla;
+    }
+
     protected static CatalogTable getTable(Cluster cluster, Feed feed) {
         // check if table is overridden in cluster
         if (cluster.getTable() != null) {

http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/dbdb7075/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
----------------------------------------------------------------------
diff --git a/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java b/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
index a724695..b4d29ee 100644
--- a/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
+++ b/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
@@ -26,12 +26,14 @@ import org.apache.falcon.entity.store.ConfigurationStore;
 import org.apache.falcon.entity.v0.Entity;
 import org.apache.falcon.entity.v0.EntityGraph;
 import org.apache.falcon.entity.v0.EntityType;
+import org.apache.falcon.entity.v0.Frequency;
 import org.apache.falcon.entity.v0.feed.ACL;
 import org.apache.falcon.entity.v0.feed.Cluster;
 import org.apache.falcon.entity.v0.feed.ClusterType;
 import org.apache.falcon.entity.v0.feed.Feed;
 import org.apache.falcon.entity.v0.feed.LocationType;
 import org.apache.falcon.entity.v0.feed.Location;
+import org.apache.falcon.entity.v0.feed.Sla;
 import org.apache.falcon.entity.v0.process.Input;
 import org.apache.falcon.entity.v0.process.Output;
 import org.apache.falcon.entity.v0.process.Process;
@@ -82,6 +84,7 @@ public class FeedEntityParser extends EntityParser<Feed> {
         validateFeedPath(feed);
         validateFeedPartitionExpression(feed);
         validateFeedGroups(feed);
+        validateFeedSLA(feed);
         validateACL(feed);
 
         // Seems like a good enough entity object for a new one
@@ -115,6 +118,40 @@ public class FeedEntityParser extends EntityParser<Feed> {
         return processes;
     }
 
+    private void validateFeedSLA(Feed feed) throws FalconException {
+        for (Cluster cluster : feed.getClusters().getClusters()) {
+            Sla clusterSla = FeedHelper.getSLAs(cluster, feed);
+            if (clusterSla != null) {
+                Frequency slaLowExpression = clusterSla.getSlaLow();
+                ExpressionHelper evaluator = ExpressionHelper.get();
+                ExpressionHelper.setReferenceDate(new Date());
+                Date slaLow = new Date(evaluator.evaluate(slaLowExpression.toString(), Long.class));
+
+                Frequency slaHighExpression = clusterSla.getSlaHigh();
+                Date slaHigh = new Date(evaluator.evaluate(slaHighExpression.toString(),
Long.class));
+
+                if (slaLow.after(slaHigh)) {
+                    throw new ValidationException("slaLow of Feed: " + slaLowExpression
+                            + "is greater than slaHigh: " + slaHighExpression
+                            + " for cluster: " + cluster.getName()
+                    );
+                }
+
+                // test that slaHigh is less than retention
+                Frequency retentionExpression = cluster.getRetention().getLimit();
+                Date retention = new Date(evaluator.evaluate(retentionExpression.toString(),
Long.class));
+                if (slaHigh.after(retention)) {
+                    throw new ValidationException("slaHigh of Feed: " + slaHighExpression
+                            + " is greater than retention of the feed: " + retentionExpression
+                            + " for cluster: " + cluster.getName()
+                    );
+                }
+
+
+            }
+        }
+    }
+
     private void validateFeedGroups(Feed feed) throws FalconException {
         String[] groupNames = feed.getGroups() != null ? feed.getGroups().split(",") : new
String[]{};
         final Storage storage = FeedHelper.createStorage(feed);

http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/dbdb7075/common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java
----------------------------------------------------------------------
diff --git a/common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java
b/common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java
index 3a9d508..8c67a7d 100644
--- a/common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java
+++ b/common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java
@@ -98,10 +98,15 @@ public class FeedEntityParserTest extends AbstractTestBase {
         assertEquals(feed.getName(), "clicks");
         assertEquals(feed.getDescription(), "clicks log");
         assertEquals(feed.getFrequency().toString(), "hours(1)");
+        assertEquals(feed.getSla().getSlaHigh().toString(), "hours(3)");
+        assertEquals(feed.getSla().getSlaLow().toString(), "hours(2)");
         assertEquals(feed.getGroups(), "online,bi");
 
         assertEquals(feed.getClusters().getClusters().get(0).getName(),
                 "testCluster");
+        assertEquals(feed.getClusters().getClusters().get(0).getSla().getSlaLow().toString(),
"hours(3)");
+        assertEquals(feed.getClusters().getClusters().get(0).getSla().getSlaHigh().toString(),
"hours(4)");
+
         assertEquals(feed.getClusters().getClusters().get(0).getType(),
                 ClusterType.SOURCE);
         assertEquals(SchemaHelper.formatDateUTC(feed.getClusters().getClusters().get(0).getValidity()
@@ -256,6 +261,27 @@ public class FeedEntityParserTest extends AbstractTestBase {
         parser.validate(feed);
     }
 
+    @Test(expectedExceptions = ValidationException.class, expectedExceptionsMessageRegExp
= "slaLow of Feed:.*")
+    public void testInvalidSlaLow() throws Exception {
+        Feed feed = parser.parseAndValidate((FeedEntityParserTest.class
+                .getResourceAsStream(FEED_XML)));
+        feed.getSla().setSlaLow(new Frequency("hours(4)"));
+        feed.getSla().setSlaHigh(new Frequency("hours(2)"));
+        parser.validate(feed);
+    }
+
+
+    @Test(expectedExceptions = ValidationException.class, expectedExceptionsMessageRegExp
= "slaHigh of Feed:.*")
+    public void testInvalidSlaHigh() throws Exception {
+        Feed feed = parser.parseAndValidate((FeedEntityParserTest.class
+                .getResourceAsStream(FEED_XML)));
+        feed.getSla().setSlaLow(new Frequency("hours(2)"));
+        feed.getSla().setSlaHigh(new Frequency("hours(10)"));
+        feed.getClusters().getClusters().get(0).getRetention().setLimit(new Frequency("hours(9)"));
+        parser.validate(feed);
+    }
+
+
     @Test
     public void testValidFeedGroup() throws FalconException, JAXBException {
         Feed feed1 = (Feed) EntityType.FEED.getUnmarshaller().unmarshal(

http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/dbdb7075/common/src/test/resources/config/feed/feed-0.1.xml
----------------------------------------------------------------------
diff --git a/common/src/test/resources/config/feed/feed-0.1.xml b/common/src/test/resources/config/feed/feed-0.1.xml
index 0269fa0..aa1998f 100644
--- a/common/src/test/resources/config/feed/feed-0.1.xml
+++ b/common/src/test/resources/config/feed/feed-0.1.xml
@@ -27,6 +27,7 @@
     <availabilityFlag>_SUCCESS</availabilityFlag>
 
     <frequency>hours(1)</frequency>
+    <sla slaLow="hours(2)" slaHigh="hours(3)"/>
     <timezone>UTC</timezone>
 
     <late-arrival cut-off="hours(6)"/>
@@ -36,6 +37,7 @@
             <validity start="2011-11-01T00:00Z" end="2011-12-31T00:00Z"/>
             <retention limit="hours(48)" action="delete"/>
             <!-- Limit can be in Time or Instances 100, Action ENUM DELETE,ARCHIVE -->
+            <sla slaLow="hours(3)" slaHigh="hours(4)"/>
             <locations>
                 <location type="data" path="/projects/falcon/clicks"/>
                 <location type="stats" path="/projects/falcon/clicksStats"/>

http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/dbdb7075/docs/src/site/twiki/EntitySpecification.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/EntitySpecification.twiki b/docs/src/site/twiki/EntitySpecification.twiki
index b0dfb7f..1e603ec 100644
--- a/docs/src/site/twiki/EntitySpecification.twiki
+++ b/docs/src/site/twiki/EntitySpecification.twiki
@@ -99,8 +99,7 @@ should be defined here.
 
 ---++ Feed Specification
 The Feed XSD specification is available here.
-a Feed defines various attributes of feed like feed location, frequency, late-arrival handling
-and retention policies.
+A Feed defines various attributes of feed like feed location, frequency, late-arrival handling
and retention policies.
 A feed can be scheduled on a cluster, once a feed is scheduled its retention and replication
process are triggered in a given cluster.
 <verbatim>
 <feed description="clicks log" name="clicks" xmlns="uri:falcon:feed:0.1"
@@ -128,6 +127,7 @@ Feed should contain one of the two storage options. Locations on File
System or
         <cluster name="test-cluster">
             <validity start="2012-07-20T03:00Z" end="2099-07-16T00:00Z"/>
             <retention limit="days(10)" action="delete"/>
+            <sla slaLow="hours(3)" slaHigh="hours(4)"/>
             <locations>
                 <location type="data" path="/hdfsDataLocation/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
                 <location type="stats" path="/projects/falcon/clicksStats" />
@@ -142,8 +142,11 @@ Validity of a feed on cluster specifies duration for which this feed
is valid on
 Retention specifies how long the feed is retained on this cluster and the action to be taken
on the feed after the expiry of retention period.
 The retention limit is specified by expression frequency(times), ex: if feed should be retained
for at least 6 hours then retention's limit="hours(6)".
 The field partitionExp contains partition tags. Number of partition tags has to be equal
to number of partitions specified in feed schema. A partition tag can be a wildcard(*), a
static string or an expression. Atleast one of the strings has to be an expression.
+sla specifies sla for the feed on this cluster. This is an optional parameter and sla can
be same or different from the
+global sla tag (mentioned outside the clusters tag ). This tag provides the user to flexibility
to have
+different sla for different clusters e.g. in case of replication. If this attribute is missing
then the default global
+sla is picked from the feed definition.
 Location specifies where the feed is available on this cluster. This is an optional parameter
and path can be same or different from the global locations tag value ( it is mentioned outside
the clusters tag ) . This tag provides the user to flexibility to have feed at different locations
on different clusters. If this attribute is missing then the default global location is picked
from the feed definition. Also the individual location tags data, stats, meta are optional.
-
 <verbatim>
  <location type="data" path="/projects/falcon/clicks" />
  <location type="stats" path="/projects/falcon/clicksStats" />
@@ -230,6 +233,17 @@ A feed has a frequency which specifies the frequency by which this feed
is gener
 ex: it can be generated every hour, every 5 minutes, daily, weekly etc.
 valid frequency type for a feed are minutes, hours, days, months. The values can be negative,
zero or positive.
 
+---+++ SLA
+<verbatim>
+    <sla slaLow="hours(40)" slaHigh="hours(44)" />
+</verbatim>
+
+A feed can have SLA and each SLA has two properties - slaLow and slaHigh. Both slaLow and
slaHigh are written using
+expressions like frequency. slaLow is intended to serve for alerting for feed instances which
are in danger of missing their
+availability SLAs. slaHigh is intended to serve for reporting the feeds which missed their
SLAs. SLAs are relative to
+feed instance time.
+
+
 ---+++ Late Arrival
 
 <verbatim>


Mime
View raw message