hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Marion (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10370) Allow DataNode to be started with numactl
Date Tue, 24 May 2016 16:24:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298453#comment-15298453

Dave Marion commented on HDFS-10370:

bq.  Could you please update the patch? 

  I'll try to get to it in the next day or so.

bq. Could you elaborate a bit more the use cases?

 I want to be able to balance memory and cpu allocation for multiple processes on a single
server. To do so I need those processes to have the ability to be managed. Specifically, I
want to run multiple Accumulo tablet servers on a single host where a DN resides. One example
is to interleave memory allocations for the DN across the numa nodes, then start 1 Accumulo
tablet server per numa node.

bq. If we are moving into the territory of numa awareness, shall we consider a solution more
generic than just Datanode?

 I'm not looking for that at the moment.

bq. Do we plan to support membind or cpubind? How to assign daemons to different numa nodes?

 The patch provides a default behavior (interleave memory), but that can be overridden such
that the user can change the numactl options

bq. How to deal with imbalance in usage?

 This is an advanced feature. I would assume that the person enabling this knows what numactl
is, how to use it, and what the side effects could be.

bq. How to support this feature across platforms?

 I'm not sure if/how other platforms support something like this.

> Allow DataNode to be started with numactl
> -----------------------------------------
>                 Key: HDFS-10370
>                 URL: https://issues.apache.org/jira/browse/HDFS-10370
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>            Reporter: Dave Marion
>            Assignee: Dave Marion
>         Attachments: HDFS-10370-1.patch, HDFS-10370-2.patch, HDFS-10370-3.patch
> Allow numactl constraints to be applied to the datanode process. The implementation I
have in mind involves two environment variables (enable and parameters) in the datanode startup
process. Basically, if enabled and numactl exists on the system, then start the java process
using it. Provide a default set of parameters, and allow the user to override the default.
Wiring this up for the non-jsvc use case seems straightforward. Not sure how this can be supported
using jsvc.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message