hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7017) Implement OutputStream for libhdfs3
Date Wed, 05 Nov 2014 18:29:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198791#comment-14198791

Colin Patrick McCabe commented on HDFS-7017:

bq. Unfortunately we are the 0.0001% users who disable memory overcommit. And we also observed
the std::bad_alloc in our stress test. So it is important to not let the library die in this
case and give the application opportunity to handle it. For instance the C API of libhdfs3
return an error flag and set errno to ENOMEM and the application will abort the query to free
the memory.

Thanks, [~wangzw].  That is an interesting data point.

Turning off memory overcommit tends not to work too well on UNIX, since when an application
tries to fork(), the memory required doubles briefly.  The new child process may never use
any of that memory reservation (and copy-on-write means the overhead may be 0), but the system
can't know that at the time the {{fork}} call is made.  Even if the next thing the process
wants to do is exec() a tiny program, a "strict no-overcommit" system (like Linux with certain
configurations) will deny the fork().  This happens a lot in Hadoop because our big Java processes
fork and exec small utility programs like "groups", "id", and so forth.  We have been gradually
adding JNI versions for all these use-cases, but some still remain.

bq. You are right about that we should write some log instead of exit the lease renewer thread
quietly. Adding another try ... catch block is a good suggestion.


[~wheat9], did you want to look at this before it gets committed?  Let me know, otherwise
I'll commit in a day or two.

> Implement OutputStream for libhdfs3
> -----------------------------------
>                 Key: HDFS-7017
>                 URL: https://issues.apache.org/jira/browse/HDFS-7017
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: Zhanwei Wang
>            Assignee: Zhanwei Wang
>         Attachments: HDFS-7017-pnative.002.patch, HDFS-7017-pnative.003.patch, HDFS-7017-pnative.004.patch,
> Implement pipeline and OutputStream C++ interface

This message was sent by Atlassian JIRA

View raw message