hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Roelofs <roel...@yahoo-inc.com>
Subject Re: Thread safety issues with JNI/native code from map tasks?
Date Sat, 29 Jan 2011 01:59:15 GMT
Todd Lipcon wrote:

> JNI may also work fine with no GC running, but then work badly when GC kicks
> in at a bad time. For example, if you grab a pointer to a String or array,
> you need to essentially lock them so the GC doesn't relocate the objects
> underneath you. For example, maybe you're releasing one of these references
> and then continuing to use it?

Excellent point!  And one I should have remembered, too.

Keith, take a look at the native ZlibCompressor interface to see one way of
handling this.  (It pins a number of buffers in memory and puts them into a
pool, if I remember Chris Douglas's explanation correctly.  I didn't need to
dive into that level of detail myself for what I was working on, so I never
touched the buffer code and might not be remembering it entirely accurately,
but that's the gist, anyway.)

On 0.20.x or 0.22/trunk:
  hadoop-common/src/native/src/org/apache/hadoop/io/compress
  hadoop-common/src/{core or java}/org/apache/hadoop/io/compress

Greg

Mime
View raw message