falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajay Yadav <ajayn...@gmail.com>
Subject Re: Relativity of validity timestamps for process & feeds
Date Mon, 06 Jul 2015 05:54:20 GMT
Hi Michael,

Short answer to your question is: *Yes this is the expected behavior*.
Actual feed instances considered depend on process instance time, EL used,
feed's start time and feed's frequency. Actually, this is not a falcon
quirk but because of how oozie resolves ELs to actual feed/process
instances.

Oozie resolves EL to a time relative to the process instance time and if
there is no feed instance corresponding to that time then the one just
before that time is picked. If you can afford to do so, then aligning your
ELs and start times of feed and processes will help in avoiding behavior
like this in lot of scenarios. So if you are using ELs like yesterday(0,0)
and today(0,0) keeping your process and feed validity to 00:00 will
definitely help as the EL will resolve to an actual time instance.

Long answer[;)]: I am writing the details of this behavior in much more
detail, with examples and will send it across as soon as I finish it.





On Mon, Jul 6, 2015 at 10:28 AM, Michael McCarthy <
michael.mccarthy@inmobi.com> wrote:

> I'm a fairly new user to Falcon - any help clarifying the following
> observations would be most welcome.
>
> Specifically, it appears that the instance selected for a feed while using
> "yesterday(0,0)" in a process spec is dependent on the relative validity
> timestamps of the feed and process.
>
> For instance, I had initially given an output feed a validity timestamp
> with the hour: 08:00 to align with the daily process generating them; which
> also has a validity timestamp hour of 08:00. The process uses
> "yesterday(0,0)" as the output feed instance.
>
> What I noticed during experimentation is that Falcon was using an output
> timestamp that was one day before the one that I wanted. Specifically,
> Falcon was using feeds for 2015-07-01 instead of the intended 2015-07-02.
> When I changed the output feed validity timestamps to 00:00, the correct
> day was used.
>
> So it appears that one needs to be careful with the hours given for feeds
> that have a daily resolution.  If the process that uses them inadvertently
> specifies an hour before or after the feed time, the incorrect day may be
> selected for the job.
>
> Is this behaviour expected? Am I correct in assuming that it is good
> practice to set validity timestamp hours for daily feeds to 00:00? Or that
> processes should specify daily feed instances as 23:00 when using ELs such
> as "yesterday" or "today"?
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message