falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Czech <eczec...@gmail.com>
Subject Re: Forced Execution of Late Input Feed Instances
Date Wed, 01 Apr 2015 16:23:19 GMT
Thanks Srikanth,

That helps for sure, though I'll probably have to delay things
unnecessarily to do that.

Would a future gating primitive in the works cover a case like this?

- We want a process instance to run right away if all data is present
- We want the process to wait up to X minutes for late data before running
anyways

I hope I'm not missing anything but it seems like with optional inputs that
I'll just have to make "right away" be some time further in the future
(i.e. the time I'd like the process to run plus the X minute delay).

Think there's any hope to pull that off with existing functionality?

Thanks again!



On Wed, Apr 1, 2015 at 11:54 AM, Srikanth Sundarrajan <sriksun@hotmail.com>
wrote:

> This is a case where one or more inputs is optional. This can be managed
> by configured the input as optional. The job in this case would run every 6
> hours with whatever input is available without actually waiting for any
> inputs. We are currently discussing about more gating primitives. Please
> see if the existing capability suffices. See "Optional Inputs" sections in
> http://falcon.apache.org/EntitySpecification.html for additional details.
>
> @dev is the right mailing list. We dont have the @user list created. This
> is to allow all the dev to help with user queries as well.
> Regards
> Srikanth Sundarrajan
>
> ----------------------------------------
> > From: eczech52@gmail.com
> > Date: Wed, 1 Apr 2015 10:28:10 -0400
> > Subject: Forced Execution of Late Input Feed Instances
> > To: dev@falcon.apache.org
> >
> > Apologies if this isn't the right place for a question like this, but I
> > thought it was worth asking before diving into the source code any more
> > (and I'm not sure where else to go).
> >
> > Is there a way to force a process to run if one or more of its required
> > input feed instances aren't yet available? By this I don't mean that the
> > input feed instances arrive late, I mean that they're not present and
> they
> > never will be (e.g. for sparse events recorded on a fairly small time
> scale
> > like seconds or minutes).
> >
> > I'd like to tie a process that runs every 6 hours to an input feed with
> an
> > hourly frequency but I'm not sure how to gracefully manage cases where
> one
> > of the input hours is missing in an expected way, without causing the
> other
> > 5 hours worth of data to be ignored.
> >
> > Is there a good place in the source code to look for where that behavior
> > would be managed (to answer the question myself perhaps)?
> >
> > Thanks and sorry again for posing a user question to a dev list!
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message