hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-14033) Reduce fair call queue lock contention
Date Fri, 27 Jan 2017 22:08:24 GMT
Daryn Sharp created HADOOP-14033:
------------------------------------

             Summary: Reduce fair call queue lock contention
                 Key: HADOOP-14033
                 URL: https://issues.apache.org/jira/browse/HADOOP-14033
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: ipc
    Affects Versions: 2.7.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


Under heavy load the call queue may run dry yet clients experience high latency.

The fcq requires producers and consumers to sync via a shared lock.  Polling consumers hold
the lock while scanning all sub-queues.  Consumers are serialized despite the sub-queues being
thread-safe blocking queues.  The effect is to cause other producers/consumers to frequently
park.

The lock is unfair, so producers/consumers attempt to barge in on the lock.  The outnumbered
producers tend to remain blocked for an extended time.  As load increases and the queues fill,
the barging consumers drain the queues faster than the producers can fill it.

Server metrics provide an illusion of healthy throughput, response time, and call queue length
due to starvation on the ingress.   Often as the load gets worse, the server looks better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message