hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6912) SharedFileDescriptorFactory should not allocate sparse files
Date Tue, 26 Aug 2014 20:37:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111284#comment-14111284

Colin Patrick McCabe commented on HDFS-6912:

bq. Hey Colin. Did you verify that tmpfs supports fallocate going back to old versions? Looking
at the kernel git history, it was only added in mid 2012 (e2d12e22c59ce714008aa5266d769f8568d74eac)
corresponding to version 3.5. So, I'm not sure if it would be supported on el6 for example
(maybe they backported it, maybe not).

I believe the glibc {{posix_fallocate}} wrapper falls back to using {{write()}} calls when
{{fallocate}} itself is not supported by the kernel.  There is some discussion here:  https://lists.gnu.org/archive/html/bug-coreutils/2009-05/msg00207.html
which talks about:

bq. i.e. fall back to using write() as the glibc posix_fallocate() implementation does.

But, I think it's simpler to just use {{write}} here.  Any performance advantage to using
{{ftruncate}} + {{fallocate}} is going to be extremely tiny (or nonexistent) since this file
is only 8192 bytes.  And {{write}} is much more portable.  So here is a new version that does

> SharedFileDescriptorFactory should not allocate sparse files
> ------------------------------------------------------------
>                 Key: HDFS-6912
>                 URL: https://issues.apache.org/jira/browse/HDFS-6912
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: caching
>    Affects Versions: 2.5.0
>         Environment: HDFS Data node, with 8 gb tmpfs in /dev/shm
>            Reporter: Gopal V
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-6912.001.patch
> SharedFileDescriptor factory should not allocate sparse files.  Sparse files can lead
to a SIGBUS later in the short-circuit reader when we try to read from the sparse file and
memory is not available.
> Note that if swap is enabled, we can still get a SIGBUS even with a non-sparse file,
since the JVM uses MAP_NORESERVE in mmap.

This message was sent by Atlassian JIRA

View raw message