hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Leung <lle...@ddn.com>
Subject RE: how to turn on NativeIO.posixFadviseIfPossible
Date Fri, 21 Jun 2013 16:19:33 GMT
This looks like a compilation problem on the native hadoop libraries.

Please locate the libhadoop.so library on your system and run
[shell]  objdump -Tt libhadoop.so | grep -I fadvise

If you don't see something like the following *the GLIBC* part  (that means the system where
the share lib was compiled did not have it)

00000000000056a0 g     F .text  00000000000000a3              Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
0000000000000000       F *UND*  0000000000000000              posix_fadvise@@GLIBC_2.2.5
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 posix_fadvise
00000000000056a0 g    DF .text  00000000000000a3  Base        Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise

Note: objdump is from binutils- rpm  (you can use yum install to install it if you don't have

-----Original Message-----
From: Jun Li [mailto:jltz922181@gmail.com] 
Sent: Friday, June 21, 2013 1:35 AM
To: user@hadoop.apache.org
Subject: how to turn on NativeIO.posixFadviseIfPossible


I downloaded the current stable version from the Apache Hadoop web site, hadoop-1.1.2. My
machine is an AMD-based machine and Redhat Enterprise 6.1. The detailed Linux kernel version

 I ran the TestNativeIO.java under the distribution directory of "test/org/apache/hadoop/io/nativeio/TestNativeIO.java"
and tried to understand how NativeIO.posixFadviseIfPossible behaves, in particular, to check
whether "posix_fadvise" is turned on or not. I am interested in this call as it is used in
Read-Ahead-Pool to cache data to the OS's buffer cache.  The following is the test case that
I ran:

  public void testPosixFadvise() throws Exception {
    FileInputStream fis = new FileInputStream("/dev/zero");
    try {
      NativeIO.posixFadviseIfPossible(fis.getFD(), 0, 0,
    } catch (NativeIOException noe) {
      // we should just skip the unit test on machines where we don't
      // have fadvise support
    } finally {

However, when I stepped into the code and reached "NativeIO.java"
under the package of "org.apache.hadoop.io.nativeio",  in the particular call below:

public static void posixFadviseIfPossible(
      FileDescriptor fd, long offset, long len, int flags)
      throws NativeIOException {

    if (nativeLoaded && fadvisePossible) {
      try {
        posix_fadvise(fd, offset, len, flags);
      } catch (UnsupportedOperationException uoe) {
        fadvisePossible = false;
      } catch (UnsatisfiedLinkError ule) {
        fadvisePossible = false;

The call to "posix_fadvise"  threw the "UnsupportedOperationException"

I further traced to the native library, and in the code "NativeIO.c", I found

  JNIEnv *env, jclass clazz,
  jobject fd_object, jlong offset, jlong len, jint flags) { #ifndef HAVE_POSIX_FADVISE
  THROW(env, "java/lang/UnsupportedOperationException",
        "fadvise support not available"); #else


I believe that the problem of throwing the exception is because "HAVE_POSIX_FADVISE" is not
defined.  I made sure that the native IO library is loaded properly in the Java code, as I
can successfully run the other test cases in "TestNativeIO.java".

So my question is: should I re-compile the "libhadoop" in order to get the version of the
shared library that can have "HAVE_POSIX_FADVISE"
turned on? Or by default, FADVISE is turned on already?

Thank you!

View raw message