hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhihai xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12189) CallQueueManager may drop elements from the queue sometimes when calling swapQueue
Date Tue, 07 Jul 2015 18:56:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617186#comment-14617186
] 

zhihai xu commented on HADOOP-12189:
------------------------------------

I uploaded a new patch HADOOP-12189.001.patch. I did some optimization in the new patch.
With the new patch, the performance is much better. it can process 377,932 elements in less
than 297ms using 120 threads for put and 120 threads for take. The performance with the new
patch is very close to the performance without patch, less than 15% difference.

> CallQueueManager may drop elements from the queue sometimes when calling swapQueue
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-12189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12189
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, test
>    Affects Versions: 2.7.1
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>         Attachments: HADOOP-12189.000.patch, HADOOP-12189.001.patch
>
>
> CallQueueManager may drop elements from the queue sometimes when calling {{swapQueue}}.

> The following test failure from TestCallQueueManager shown some elements in the queue
are dropped.
> https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/
> {code}
> java.lang.AssertionError: expected:<27241> but was:<27245>
> 	at org.junit.Assert.fail(Assert.java:88)
> 	at org.junit.Assert.failNotEquals(Assert.java:743)
> 	at org.junit.Assert.assertEquals(Assert.java:118)
> 	at org.junit.Assert.assertEquals(Assert.java:555)
> 	at org.junit.Assert.assertEquals(Assert.java:542)
> 	at org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
> {code}
> It looked like the elements in the queue are dropped due to {{CallQueueManager#swapQueue}}
> Looked at the implementation of {{CallQueueManager#swapQueue}}, there is a possibility
that the elements in the queue are dropped. If the queue is full, the calling thread for {{CallQueueManager#put}}
is blocked for long time. It may put the element into the old queue after queue in {{takeRef}}
is changed by swapQueue, then this element in the old queue will be dropped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message