apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Deshmukh <sand...@datatorrent.com>
Subject Re: Reg. file splitter blocks
Date Wed, 27 Jan 2016 17:14:10 GMT
You can use blocksThreshold for this purpose. Here is the relevant java doc

/**
* This is a threshold on the no. of blocks emitted per window. A lot of
blocks emitted
* per window can overwhelm the downstream operators. This setting helps to
control that.
*/
@Min(1)
protected int blocksThreshold;

Source:
https://github.com/apache/incubator-apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/io/fs/FileSplitter.java

Regards,
Sandeep

On Wed, Jan 27, 2016 at 10:36 PM, Kottapalli, Venkatesh <
VKottapalli@directv.com> wrote:

> Hi,
>
>                 Is there a way to limit the number of blocks that
> filesplitter emits per window?
>
>                 I have a custom reader that would read data from the
> blocks, I am looking for a way to adjust the speed of filesplitter to that
> of the reader so that reader would require less resources to process the
> blocks.
>
> Please suggest.
>
> -Venkatesh.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message