hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Li <jltz922...@gmail.com>
Subject how to turn on NativeIO.posixFadviseIfPossible
Date Fri, 21 Jun 2013 08:35:03 GMT

I downloaded the current stable version from the Apache Hadoop web
site, hadoop-1.1.2. My machine is an AMD-based machine and Redhat
Enterprise 6.1. The detailed Linux kernel version is:

 I ran the TestNativeIO.java under the distribution directory of
"test/org/apache/hadoop/io/nativeio/TestNativeIO.java" and tried to
understand how NativeIO.posixFadviseIfPossible behaves, in particular,
to check whether "posix_fadvise" is turned on or not. I am interested
in this call as it is used in Read-Ahead-Pool to cache data to the
OS's buffer cache.  The following is the test case that I ran:

  public void testPosixFadvise() throws Exception {
    FileInputStream fis = new FileInputStream("/dev/zero");
    try {
      NativeIO.posixFadviseIfPossible(fis.getFD(), 0, 0,
    } catch (NativeIOException noe) {
      // we should just skip the unit test on machines where we don't
      // have fadvise support
    } finally {

However, when I stepped into the code and reached "NativeIO.java"
under the package of "org.apache.hadoop.io.nativeio",  in the
particular call below:

public static void posixFadviseIfPossible(
      FileDescriptor fd, long offset, long len, int flags)
      throws NativeIOException {

    if (nativeLoaded && fadvisePossible) {
      try {
        posix_fadvise(fd, offset, len, flags);
      } catch (UnsupportedOperationException uoe) {
        fadvisePossible = false;
      } catch (UnsatisfiedLinkError ule) {
        fadvisePossible = false;

The call to "posix_fadvise"  threw the "UnsupportedOperationException"

I further traced to the native library, and in the code "NativeIO.c", I found

  JNIEnv *env, jclass clazz,
  jobject fd_object, jlong offset, jlong len, jint flags)
  THROW(env, "java/lang/UnsupportedOperationException",
        "fadvise support not available");


I believe that the problem of throwing the exception is because
"HAVE_POSIX_FADVISE" is not defined.  I made sure that the native IO
library is loaded properly in the Java code, as I can successfully run
the other test cases in "TestNativeIO.java".

So my question is: should I re-compile the "libhadoop" in order to get
the version of the shared library that can have "HAVE_POSIX_FADVISE"
turned on? Or by default, FADVISE is turned on already?

Thank you!

View raw message