Mailing-List: contact commits-help@metron.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@metron.incubator.apache.org
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: mattf@apache.org
To: commits@metron.incubator.apache.org
Message-Id: <ec601282ebf54fe6b0553514a1668265@git.apache.org>
Subject: incubator-metron git commit: METRON-764 DST bug in
 metron-profiler-client Unit Tests (mattf-horton) closes
 apache/incubator-metron#476
Date: Fri, 17 Mar 2017 00:37:09 +0000 (UTC)
archived-at: Fri, 17 Mar 2017 00:37:15 -0000

Repository: incubator-metron
Updated Branches:
  refs/heads/master 68a334a87 -> f1435a6a7


METRON-764 DST bug in metron-profiler-client Unit Tests (mattf-horton) closes apache/incubator-metron#476


Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/f1435a6a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/f1435a6a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/f1435a6a

Branch: refs/heads/master
Commit: f1435a6a75f016294920ac328ba79474fbf129e1
Parents: 68a334a
Author: mattf-horton <mfoley@hortonworks.com>
Authored: Thu Mar 16 17:14:52 2017 -0700
Committer: YOUR NAME as In Apache <YOU@apache.org>
Committed: Thu Mar 16 17:14:52 2017 -0700

----------------------------------------------------------------------
 .../metron-profiler-client/README.md            | 29 ++++++++++++++++++++
 .../client/window/WindowProcessorTest.java      | 13 +++++++++
 2 files changed, 42 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f1435a6a/metron-analytics/metron-profiler-client/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/README.md b/metron-analytics/metron-profiler-client/README.md
index 656bb25..aa35136 100644
--- a/metron-analytics/metron-profiler-client/README.md
+++ b/metron-analytics/metron-profiler-client/README.md
@@ -253,6 +253,35 @@ The specifiers are a set of fixed specifiers available as part of the language:
     * Countries supported are those supported in [jollyday](https://github.com/svendiedrichsen/jollyday/tree/master/src/main/resources/holidays)
     * Example: `holiday:us:nyc` would be the holidays of New York City, USA
     * Example: `holiday:hu` would be the holidays of Hungary
+    
+**WARNING: Daylight Savings Time effects**
+
+While Universal Time (UTC) is nice and constant, many servers are set to local timezones that enable Daylight Savings Time (DST).
+This means that twice a year, on DST transition weekends, "Sunday" is either 23 or 25 hours long.  However, durations specified
+as "7 days ago" are always interpreted as "7*24 hours ago".  This can lead to some surprising effects when using days of the week
+as inclusion or exclusion specifiers.
+
+For example, the profile window specified by the phrase "30 minute window every 24 hours from 7 days ago"
+will always have 7 thirty-minute intervals, and these will normally occur on 5 weekdays and 2 weekend days.
+However, if you invoke this window at 12:15am any day during the week following the start of DST, you will get
+these intervals (supposing you start early on a Wednesday morning):
+```
+Tuesday 12:15am-12:45am (yesterday)
+Monday 12:15am-12:45am
+Saturday 11:15pm-11:45pm (skipped Sunday!)
+Friday 11:15pm-11:45pm
+Thursday 11:15pm-11:45pm
+Wednesday 11:15pm-11:45pm
+Tuesday 11:15pm-11:45pm
+```
+
+Sunday got skipped over because it was only 23 hours long; that is, there were 24 hours between Saturday 11:15pm and Monday 12:15am.
+So if you specified "excluding weekends", you would get 6 days' intervals instead of the expected 5.  There are multiple variations
+on this theme.
+
+Remember that the underlying time is kept in UTC, so the data is always correct.  It is only when attempting to interpret UTC as
+local time, date, and day, that these confusions may occur.  They may be eliminated by setting your server timezone to UTC, or otherwise
+disabling DST.
      
 **Examples**
 

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f1435a6a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java
index 94f9e04..79212e2 100644
--- a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java
+++ b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java
@@ -29,6 +29,13 @@ import java.util.Date;
 import java.util.List;
 import java.util.concurrent.TimeUnit;
 
+/**
+ * On any test case where we explicitly include or exclude days of the week, in 24hr periods,
+ * we need to understand that on Daylight Savings Time (DST) transition weekends,
+ * Sunday is either 23 or 25 hours long.  This leads to surprising correct results,
+ * that may fail the assert criteria, if one starts the hour before or after midnight.
+ * Thus in such test cases, we force now.setHours(6).
+ */
 public class WindowProcessorTest {
 
   @Test
@@ -129,6 +136,7 @@ public class WindowProcessorTest {
     Gotta be 2 tuesdays in 14 days.
      */
       Date now = new Date();
+      now.setHours(6); //avoid DST impacts if near Midnight
       List<Range<Long>> intervals = w.toIntervals(now.getTime());
       Assert.assertEquals(2, intervals.size());
     }
@@ -140,6 +148,7 @@ public class WindowProcessorTest {
     Gotta be 2 days with the same dow in 14 days.
      */
       Date now = new Date();
+      now.setHours(6); //avoid DST impacts if near Midnight
       List<Range<Long>> intervals = w.toIntervals(now.getTime());
       Assert.assertEquals(2, intervals.size());
     }
@@ -162,6 +171,7 @@ public class WindowProcessorTest {
     Window w = WindowProcessor.process("30 minute window every 24 hours from 7 days ago including saturdays excluding weekends");
 
     Date now = new Date();
+    now.setHours(6); //avoid DST impacts if near Midnight
     List<Range<Long>> intervals = w.toIntervals(now.getTime());
     Assert.assertEquals(0, intervals.size());
   }
@@ -171,6 +181,7 @@ public class WindowProcessorTest {
     Window w = WindowProcessor.process("30 minute window every 24 hours from 7 days ago excluding weekends");
 
     Date now = new Date();
+    now.setHours(6); //avoid DST impacts if near Midnight
     List<Range<Long>> intervals = w.toIntervals(now.getTime());
     Assert.assertEquals(5, intervals.size());
   }
@@ -198,6 +209,7 @@ public class WindowProcessorTest {
       Window w = WindowProcessor.process("1 hour window every 24 hours starting from 56 days ago including this day of the week");
 
       Date now = new Date();
+      now.setHours(6); //avoid DST impacts if near Midnight
       List<Range<Long>> intervals = w.toIntervals(now.getTime());
       Assert.assertEquals(8, intervals.size());
     }
@@ -208,6 +220,7 @@ public class WindowProcessorTest {
     Window w = WindowProcessor.process("30 minute window every 24 hours from 7 days ago excluding weekdays");
 
     Date now = new Date();
+    now.setHours(6); //avoid DST impacts if near Midnight
     List<Range<Long>> intervals = w.toIntervals(now.getTime());
     Assert.assertEquals(2, intervals.size());
   }