apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhupesh Chawda <bhup...@datatorrent.com>
Subject Re: Restricting emit speed of AbstractFileInputOperator.
Date Sat, 08 Apr 2017 01:53:39 GMT
I think the understanding is wrong.

Platform calls emitTuples multiple times in a window. This number is
unknown; it depends on the window time. We can limit this to x.

emitBatchSize controls the number of tuples emitted in one such call. Set
this to 1.

This should result in at most x tuples per window. Note that it can be less
than x as well.

~ Bhupesh




On Apr 8, 2017 01:56, "Ambarish Pande" <ambarish.pande2011@gmail.com> wrote:

Hello Bhupesh Sir,
But does that mean I am emitting only 'x' lines from the file?. Because
from what I understood, emitTuples() emits multiple lines in a single call
and emitBatchSize controls number of times emitTuples is called in a
window. Am I right?. I inferred this from the following

The platform invokes the emitTuples() callback multiple time in each
> streaming window; within a single such call, if a large number of tuples
> are emitted, there is some risk that they may overwhelm the downstream
> operators especially if they are performing some compute intensive
> operation.


Thank You.

On Fri, Apr 7, 2017 at 1:44 PM, Bhupesh Chawda <bhupesh@datatorrent.com>
wrote:

> You can set emitBatchSize to 1 and make sure emitTuples is called just 'x'
> times within a window. You can do this manually by keeping a count and
> resetting it in beginWindow().
>
> ~ Bhupesh
>
>
> _______________________________________________________
>
> Bhupesh Chawda
>
> E: bhupesh@datatorrent.com | Twitter: @bhupeshsc
>
> www.datatorrent.com  |  apex.apache.org
>
>
>
> On Fri, Apr 7, 2017 at 1:38 PM, Ambarish Pande <
> ambarish.pande2011@gmail.com> wrote:
>
>> Yes i tried. That just gives me control on how many times emitTuples is
>> called. I want control on number of tuples emitted.
>>
>> Thank you.
>>
>> Sent from my iPhone
>>
>> On 07-Apr-2017, at 8:08 AM, Yogi Devendra <devendra.vyavahare@gmail.com>
>> wrote:
>>
>> Have you tried *emitBatchSize *as mentioned https://apex.apache.
>> org/docs/malhar/operators/fsInputOperator/
>>
>> ~ Yogi
>>
>> On 3 April 2017 at 00:05, Ambarish Pande <ambarish.pande2011@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> How can i make the AbstractFileInputOperator emit only 'x' number of
>>> lines per window. Is there a hook for that. Or i have to do it manually?
>>>
>>> Thank You.
>>>
>>
>>
>

Mime
View raw message