hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9194) RPC Support for QoS
Date Fri, 11 Jan 2013 19:56:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551466#comment-13551466

Colin Patrick McCabe commented on HADOOP-9194:

Interesting stuff, guys.  many good points have been brought up.

The code path for UNIX domain sockets is like this now:
1. server calls accept(), gets a socket, hands it off to worker thread
2. worker thread reads the RPC header to find the type of message and length
3. worker thread reads the message
4. worker thread processes the message

Having QoS information in the header would allow us to prioritize the message after step #2.
Having QoS information in the protobuf would allow us to prioritize the message after step

Since messages are normally just a few bytes, I'm not sure that this would be a big win.

In general, I think using a separate UNIX domain socket would probably make more sense.  It
would also allow us to use operating system features like the accept backlog to our advantage--
when using a single socket, we have to implement all that ourselves, and we don't really have
the tools in userspace to do a good job.
> RPC Support for QoS
> -------------------
>                 Key: HADOOP-9194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 2.0.2-alpha
>            Reporter: Luke Lu
> One of the next frontiers of Hadoop performance is QoS (Quality of Service). We need
QoS support to fight the inevitable "buffer bloat" (including various queues, which are probably
necessary for throughput) in our software stack. This is important for mixed workload with
different latency and throughput requirements (e.g. OLTP vs OLAP, batch and even compaction
I/O) against the same DFS.
> Any potential bottleneck will need to be managed by QoS mechanisms, starting with RPC.

> How about adding a one byte DS (differentiated services) field (a la the 6-bit DS field
in IP header) in the RPC header to facilitate the QoS mechanisms (in separate JIRAs)? The
byte at a fixed offset (how about 0?) of the header is helpful for implementing high performance
QoS mechanisms in switches (software or hardware) and servers with minimum decoding effort.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message