apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ambarish Pande <ambarish.pande2...@gmail.com>
Subject Re: Restricting emit speed of AbstractFileInputOperator.
Date Sat, 08 Apr 2017 05:56:42 GMT
Oh Okay. i couldn't find this mentioned clearly in the docs, that is why I
misinterpreted it.
Thank You.

On Sat, Apr 8, 2017 at 7:23 AM, Bhupesh Chawda <bhupesh@datatorrent.com>
wrote:

> I think the understanding is wrong.
>
> Platform calls emitTuples multiple times in a window. This number is
> unknown; it depends on the window time. We can limit this to x.
>
> emitBatchSize controls the number of tuples emitted in one such call. Set
> this to 1.
>
> This should result in at most x tuples per window. Note that it can be
> less than x as well.
>
> ~ Bhupesh
>
>
>
>
> On Apr 8, 2017 01:56, "Ambarish Pande" <ambarish.pande2011@gmail.com>
> wrote:
>
> Hello Bhupesh Sir,
> But does that mean I am emitting only 'x' lines from the file?. Because
> from what I understood, emitTuples() emits multiple lines in a single call
> and emitBatchSize controls number of times emitTuples is called in a
> window. Am I right?. I inferred this from the following
>
> The platform invokes the emitTuples() callback multiple time in each
>> streaming window; within a single such call, if a large number of tuples
>> are emitted, there is some risk that they may overwhelm the downstream
>> operators especially if they are performing some compute intensive
>> operation.
>
>
> Thank You.
>
> On Fri, Apr 7, 2017 at 1:44 PM, Bhupesh Chawda <bhupesh@datatorrent.com>
> wrote:
>
>> You can set emitBatchSize to 1 and make sure emitTuples is called just
>> 'x' times within a window. You can do this manually by keeping a count and
>> resetting it in beginWindow().
>>
>> ~ Bhupesh
>>
>>
>> _______________________________________________________
>>
>> Bhupesh Chawda
>>
>> E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
>>
>> www.datatorrent.com  |  apex.apache.org
>>
>>
>>
>> On Fri, Apr 7, 2017 at 1:38 PM, Ambarish Pande <
>> ambarish.pande2011@gmail.com> wrote:
>>
>>> Yes i tried. That just gives me control on how many times emitTuples is
>>> called. I want control on number of tuples emitted.
>>>
>>> Thank you.
>>>
>>> Sent from my iPhone
>>>
>>> On 07-Apr-2017, at 8:08 AM, Yogi Devendra <devendra.vyavahare@gmail.com>
>>> wrote:
>>>
>>> Have you tried *emitBatchSize *as mentioned https://apex.apache.
>>> org/docs/malhar/operators/fsInputOperator/
>>>
>>> ~ Yogi
>>>
>>> On 3 April 2017 at 00:05, Ambarish Pande <ambarish.pande2011@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> How can i make the AbstractFileInputOperator emit only 'x' number of
>>>> lines per window. Is there a hook for that. Or i have to do it manually?
>>>>
>>>> Thank You.
>>>>
>>>
>>>
>>
>
>

Mime
View raw message