Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of lleung@ddn.com designates
 64.47.133.206 as permitted sender)
From: Leo Leung <lleung@ddn.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: RE: how to turn on NativeIO.posixFadviseIfPossible
Thread-Topic: how to turn on NativeIO.posixFadviseIfPossible
Thread-Index: AQHOblpXenw9u/19z0+7w7BdbSjlNJlAWG5w
Date: Fri, 21 Jun 2013 16:19:33 +0000
Message-ID: 
 <1C40D33AEC9DCA40A06F4637B6E58ABF46ABD19B@LAX-EX-MB2.datadirect.datadirectnet.com>
References: 
 <CACu+R8L6c6U2=+ch8sLN-1GJ5uH+PA+_6bfPsS3U2Rjj=LhE7w@mail.gmail.com>
In-Reply-To: 
 <CACu+R8L6c6U2=+ch8sLN-1GJ5uH+PA+_6bfPsS3U2Rjj=LhE7w@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

This looks like a compilation problem on the native hadoop libraries.

Please locate the libhadoop.so library on your system and run
[shell]  objdump -Tt libhadoop.so | grep -I fadvise

If you don't see something like the following *the GLIBC* part  (that means=
 the system where the share lib was compiled did not have it)

00000000000056a0 g     F .text  00000000000000a3              Java_org_apac=
he_hadoop_io_nativeio_NativeIO_posix_1fadvise
0000000000000000       F *UND*  0000000000000000              posix_fadvise=
@@GLIBC_2.2.5
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 posix_fadvise
00000000000056a0 g    DF .text  00000000000000a3  Base        Java_org_apac=
he_hadoop_io_nativeio_NativeIO_posix_1fadvise

Note: objdump is from binutils- rpm  (you can use yum install to install it=
 if you don't have it)


-----Original Message-----
From: Jun Li [mailto:jltz922181@gmail.com]=20
Sent: Friday, June 21, 2013 1:35 AM
To: user@hadoop.apache.org
Subject: how to turn on NativeIO.posixFadviseIfPossible

Hi,

I downloaded the current stable version from the Apache Hadoop web site, ha=
doop-1.1.2. My machine is an AMD-based machine and Redhat Enterprise 6.1. T=
he detailed Linux kernel version is:
2.6.32-131.0.15.el6.x86_64

 I ran the TestNativeIO.java under the distribution directory of "test/org/=
apache/hadoop/io/nativeio/TestNativeIO.java" and tried to understand how Na=
tiveIO.posixFadviseIfPossible behaves, in particular, to check whether "pos=
ix_fadvise" is turned on or not. I am interested in this call as it is used=
 in Read-Ahead-Pool to cache data to the OS's buffer cache.  The following =
is the test case that I ran:

@Test
  public void testPosixFadvise() throws Exception {
    FileInputStream fis =3D new FileInputStream("/dev/zero");
    try {
      NativeIO.posixFadviseIfPossible(fis.getFD(), 0, 0,
                             NativeIO.POSIX_FADV_SEQUENTIAL);
    } catch (NativeIOException noe) {
      // we should just skip the unit test on machines where we don't
      // have fadvise support
      assumeTrue(false);
    } finally {
      fis.close();
    }

However, when I stepped into the code and reached "NativeIO.java"
under the package of "org.apache.hadoop.io.nativeio",  in the particular ca=
ll below:

public static void posixFadviseIfPossible(
      FileDescriptor fd, long offset, long len, int flags)
      throws NativeIOException {

    if (nativeLoaded && fadvisePossible) {
      try {
        posix_fadvise(fd, offset, len, flags);
      } catch (UnsupportedOperationException uoe) {
        fadvisePossible =3D false;
      } catch (UnsatisfiedLinkError ule) {
        fadvisePossible =3D false;
      }
    }
  }

The call to "posix_fadvise"  threw the "UnsupportedOperationException"
exception.

I further traced to the native library, and in the code "NativeIO.c", I fou=
nd

JNIEXPORT void JNICALL
Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise(
  JNIEnv *env, jclass clazz,
  jobject fd_object, jlong offset, jlong len, jint flags) { #ifndef HAVE_PO=
SIX_FADVISE
  THROW(env, "java/lang/UnsupportedOperationException",
        "fadvise support not available"); #else

...
}

I believe that the problem of throwing the exception is because "HAVE_POSIX=
_FADVISE" is not defined.  I made sure that the native IO library is loaded=
 properly in the Java code, as I can successfully run the other test cases =
in "TestNativeIO.java".

So my question is: should I re-compile the "libhadoop" in order to get the =
version of the shared library that can have "HAVE_POSIX_FADVISE"
turned on? Or by default, FADVISE is turned on already?

Thank you!