apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vivek Bhide <bhide.vi...@gmail.com>
Subject Exceeded allowed memory block for AvroFileInputOperator
Date Fri, 26 May 2017 14:17:04 GMT
I have a AvroFileInputOperator which has large no of files to be read when
the application starts. While the fire reading goes well at the start, I
start getting below warning in log after reading around 50+ files

Is there something to be worried about and is there a way to mitigate it?

2017-05-26 09:10:36,233 INFO  fs.AbstractFileInputOperator
(AbstractFileInputOperator.java:openFile(796)) - opening file
hdfs:///<directory_location>//partition_date=2017-05-05/000000_0_copy_28
2017-05-26 09:11:32,145 WARN  internal.DataList (DataList.java:run(743)) -
Exceeded allowed memory block allocation by 1
2017-05-26 09:11:58,595 WARN  internal.DataList (DataList.java:run(743)) -
Exceeded allowed memory block allocation by 1

Regards
Vivek



--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Exceeded-allowed-memory-block-for-AvroFileInputOperator-tp1651.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.

Mime
View raw message