apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Gupta <gau...@datatorrent.com>
Subject Re: Best way to trigger start reading directory
Date Thu, 03 Dec 2015 06:41:37 GMT
Isha,

I know if you add input port it becomes generic operator and for that you can use IdleTimeHandler
and in handleIdleTime call emitTuples only if you have received trigger from 1st reader. Hope
that helps. 

Thanks
- Gaurav

> On Dec 2, 2015, at 10:34 PM, Isha Arkatkar <isha@datatorrent.com> wrote:
> 
> Hey Gaurav,
> 
>  No that does not work I am afraid. I tried the same thing first. But when
> you have a connected input port even if it is for an Input operator, the
> Operator type changes from INPUT to GENERIC.  emitTuples is invoked in loop
> only for InputNode.. So, operator emits nothing if I add a connected input
> stream to it.
> I think, though, this might be nice to have if sending triggers between
> input operators is a common observed pattern.
> 
> I will try out the StatsListener approach.
> 
> Thanks,
> Isha
> 
> 
> 
> 
> 
> On Wed, Dec 2, 2015 at 10:10 PM, Gaurav Gupta <gaurav@datatorrent.com>
> wrote:
> 
>> Can this not be done as follows
>> 
>> 2nd reader has an input port which is connected to output port of 1st
>> reader. Once the 1st reader is done reading it can send trigger to 2nd
>> reader over the output port and 2nd reader starts reading once it gets
>> trigger.
>> 
>> Thanks
>> - Gaurav
>> 
>>> On Dec 2, 2015, at 7:19 PM, Sandeep Deshmukh <sandeep@datatorrent.com>
>> wrote:
>>> 
>>> You use StatsListener shared between two operators to trigger this.
>>> 
>>> Regards,
>>> Sandeep
>>> 
>>> On Thu, Dec 3, 2015 at 8:08 AM, Isha Arkatkar <isha@datatorrent.com>
>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>>   Yes, that should work. But suppose first operator rolled back to
>>>> previous checkpoint due to some fail over, the state of application
>> would
>>>> be reset. Except the part that the 2nd file reader which was not
>> emitting
>>>> anything in those windows, now will continue to emit tuples. That will
>> make
>>>> the state inconsistent.
>>>> May be I can create the link from downstream operator after first
>> reader is
>>>> done to handle that.
>>>> 
>>>> Thanks,
>>>> Isha
>>>> 
>>>> On Wed, Dec 2, 2015 at 5:02 PM, Munagala Ramanath <ram@datatorrent.com>
>>>> wrote:
>>>> 
>>>>> One way (a bit hacky) is to have the 2nd operator monitor an empty
>>>>> directory. Then, when the
>>>>> 1st is done reading, it sends a control tuple to a "file-link"
>> operator;
>>>>> that operator creates
>>>>> symbolic links from the directory monitored by the 2nd operator to the
>>>>> actual input files.
>>>>> That should then trigger the 2nd input operator to do its thing.
>>>>> 
>>>>> Ram
>>>>> 
>>>>> On Wed, Dec 2, 2015 at 4:43 PM, Isha Arkatkar <isha@datatorrent.com>
>>>>> wrote:
>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>>  I have an application with 2 input file reader operators. In this
>>>>> case,
>>>>>> want to trigger start reading from 2nd input location, only after
1st
>>>>>> operator is done reading. What is the best way to do this?
>>>>>> 
>>>>>> Thanks!
>>>>>> Isha
>>>>>> 
>>>>> 
>>>> 
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message