hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Zhuge (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-10370) Allow DataNode to be started with numactl
Date Mon, 30 May 2016 23:28:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306974#comment-15306974
] 

John Zhuge edited comment on HDFS-10370 at 5/30/16 11:27 PM:
-------------------------------------------------------------

Thanks [~dlmarion] for uploading the new patches.

Comments for file {{hdfs}} in {{HDFS-10370-branch-2.004.patch}}.
* 151-158: How about replacing the block with
{code}
    if [[ ${HADOOP_DN_NUMACTL_ENABLE} == true ]] ; then
        NUMACTL_ARGS=${HADOOP_DN_NUMACTL_ARGS:-"--interleave=all"}
        NUMACTL_CMD="$(command -v numactl)"
    fi
{code}
* Line 154: It is discouraged to use {{which}} to detect executable. See explanation in http://stackoverflow.com/questions/592620/check-if-a-program-exists-from-a-bash-script/677212#677212.
* 315 and 317: These 2 lines are almost identical, any way to remove redundancy?
* 317: Where is {{NUMACTL_ARGS}} initialized? Is it supposed to take the value from {{HADOOP_DN_NUMACTL_ARGS}}?
* 317: Inconsistent prefix between {{NUMA_CMD}} and {{NUMACTL_ARGS}}

[~aw] Any generic way to run numactl for Hadoop daemons with different numactl args? How to
suppot JSVC? Maybe using JNI {{set_mempolicy}}?



was (Author: jzhuge):
Thanks [~dlmarion] for uploading the new patches.

Comments for file {{hdfs}} in {{HDFS-10370-branch-2.004.patch}}.
* 151-158: How about replacing the block with
{code}
    if [[ ${HADOOP_DN_NUMACTL_ENABLE == true ]] ; then
        NUMACTL_ARGS=${HADOOP_DN_NUMACTL_ARGS:-"--interleave=all"}
        NUMACTL_CMD=$(command -v numactl)
    fi
{code}
* Line 154: It is discouraged to use {{which}} to detect executable. See explanation in http://stackoverflow.com/questions/592620/check-if-a-program-exists-from-a-bash-script/677212#677212.
* 315 and 317: These 2 lines are almost identical, any way to remove redundancy?
* 317: Where is {{NUMACTL_ARGS}} initialized? Is it supposed to take the value from {{HADOOP_DN_NUMACTL_ARGS}}?
* 317: Inconsistent prefix between {{NUMA_CMD}} and {{NUMACTL_ARGS}}

[~aw] Any generic way to run numactl for Hadoop daemons with different numactl args? How to
suppot JSVC? Maybe using JNI {{set_mempolicy}}?


> Allow DataNode to be started with numactl
> -----------------------------------------
>
>                 Key: HDFS-10370
>                 URL: https://issues.apache.org/jira/browse/HDFS-10370
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>            Reporter: Dave Marion
>            Assignee: Dave Marion
>         Attachments: HDFS-10370-1.patch, HDFS-10370-2.patch, HDFS-10370-3.patch, HDFS-10370-branch-2.004.patch,
HDFS-10370.004.patch
>
>
> Allow numactl constraints to be applied to the datanode process. The implementation I
have in mind involves two environment variables (enable and parameters) in the datanode startup
process. Basically, if enabled and numactl exists on the system, then start the java process
using it. Provide a default set of parameters, and allow the user to override the default.
Wiring this up for the non-jsvc use case seems straightforward. Not sure how this can be supported
using jsvc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message