avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ey-chih chow <eyc...@hotmail.com>
Subject RE: avro object reuse
Date Wed, 01 Jun 2011 00:40:05 GMT

I actually looked into Avro code to find out how Avro does object reuse.  I looked at AvroUtf8InputFormat
and got the following question.  Why a new Utf8 object has to be created each time the method
next(AvroWrapper<Utf8> key, NullWritable value) is called ?  Will this eat up too much
memory when we call next(key, value) many times?  Since Utf8 is mutable, can we just create
one Utf8 object for all the calls to next(key, value)?  Will this save memory?  Thanks.
Ey-Chih Chow 

From: eychih@hotmail.com
To: user@avro.apache.org
Subject: avro object reuse
Date: Tue, 31 May 2011 10:38:39 -0700

We have several mapreduce jobs using avro.  They take too much memory when running on production.
 Can anybody suggest some object reuse techniques to cut down memory usage?  Thanks.
Ey-Chih Chow 		 	   		  
View raw message