apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vlad Rozov <v.ro...@datatorrent.com>
Subject Re: Exceeded allowed memory block for AvroFileInputOperator
Date Tue, 30 May 2017 14:34:39 GMT
The warning means that the downstream operator(s) is not capable to keep 
up with the upstream operator and tuples are accumulated in the upstream 
buffer server leading to the increased latency. If the latency is within 
SLA, you may ignore the warning, otherwise try to partition downstream 
or decrease emit rate.

Thank you,

Vlad

On 5/26/17 07:17, Vivek Bhide wrote:
> I have a AvroFileInputOperator which has large no of files to be read when
> the application starts. While the fire reading goes well at the start, I
> start getting below warning in log after reading around 50+ files
>
> Is there something to be worried about and is there a way to mitigate it?
>
> 2017-05-26 09:10:36,233 INFO  fs.AbstractFileInputOperator
> (AbstractFileInputOperator.java:openFile(796)) - opening file
> hdfs:///<directory_location>//partition_date=2017-05-05/000000_0_copy_28
> 2017-05-26 09:11:32,145 WARN  internal.DataList (DataList.java:run(743)) -
> Exceeded allowed memory block allocation by 1
> 2017-05-26 09:11:58,595 WARN  internal.DataList (DataList.java:run(743)) -
> Exceeded allowed memory block allocation by 1
>
> Regards
> Vivek
>
>
>
> --
> View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Exceeded-allowed-memory-block-for-AvroFileInputOperator-tp1651.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.


Mime
View raw message