hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7753) Support fadvise and sync_data_range in NativeIO, add ReadaheadPool class
Date Wed, 26 Oct 2011 22:09:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136476#comment-13136476
] 

Todd Lipcon commented on HADOOP-7753:
-------------------------------------

bq. configure.ac, why the change to ldflags?
The issue is that the AC_CHECK_LIB macro ends up adding {{-ljvm}} to {{$LIBS}} when successful.
Then all of the following AC_CHECK_FUNCS tests were failing like so:

{code}
configure:11586: gcc -o conftest -g -O2 -I/usr/local/include -L/usr/local/lib conftest.c -ljvm
-ldl  >&5
/usr/bin/ld: cannot find -ljvm
collect2: ld returned 1 exit status
configure:11586: $? = 1
{code}

so it was failing to properly set {{HAVE_POSIX_FADVISE}}.

bq. Did you happen to test sync_file_range on both 32 and 64 bit architectures with files
greater than 4G

Nope. I unfortunately did not have the chance to test this. But I think if we commit this
as an "experimental" feature to start off, then we can address some issues like this down
the road. If your team has time to test it, that would be excellent.

bq. I was wondering if cancel should go ahead and remove the req from the workq.
It seemed like extra complexity for little gain. In watching this code run with terasorts,
etc, the queue was almost always empty (ie most of the time at least one of the RA threads
was inactive). If it's OK, I'd prefer to leave as is.

bq. Regarding the cancel race. It's not guaranteed to return EBADF because the fd is likely
to get reused immediately for something else
True. Let me add a comment as you described.

bq. It would be nice if ReadaheadRequest sanity checked curPos and maxOffsetToRead as well
bq. Might be nice in the maxOffsetToRead param to indicate what the "don't care" value is


Good ideas. Will add.
                
> Support fadvise and sync_data_range in NativeIO, add ReadaheadPool class
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-7753
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7753
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io, native
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7753.txt, hadoop-7753.txt
>
>
> This JIRA adds JNI wrappers for sync_data_range and posix_fadvise. It also implements
a ReadaheadPool class for future use from HDFS and MapReduce.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message