hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
Date Fri, 08 Jul 2016 02:13:11 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367098#comment-15367098
] 

Mingliang Liu edited comment on HADOOP-13351 at 7/8/16 2:12 AM:
----------------------------------------------------------------

{quote}
the OS declining to set a large send buffer size for a socket.
{quote}
Thanks [~fabbri] for the explanation.  In {{StandardSocketOptions}}, an attempt to set the
socket send buffer to larger than its maximum size causes it to be set to its maximum size
(refer to [link here|https://docs.oracle.com/javase/7/docs/api/java/net/StandardSocketOptions.html#SO_SNDBUF]).
I think the value for {{Socket}} would work in the same way. This way, a larger value like
512KB would not cause problems for set operation. However, the test may still fail since the
maximum size, as the default size, is system dependent.

That being said, testing the specified value is indeed tricky. To make the test assertion
covers this case, how about comparing the hinted buffer size? I see very limited chance that
the test fails, but it does confirm that, the config key works as expected.

Sample code as following:
{code:java}
  @Test
  public void testSpecifiedSendBufferSize() throws IOException {
    conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 256 * 1024); // a big hint
    final Socket s1 = createSocket();
    conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 1024);          // a small hint
    final Socket s2 = createSocket();
    LOG.info("The real big buffer size is {}, small buffer size is {}",
        s1.getSendBufferSize(), s2.getSendBufferSize());
    assertTrue(s1.getSendBufferSize() > s2.getSendBufferSize());
}
{code}

What's your thought?


was (Author: liuml07):
{quote}
the OS declining to set a large send buffer size for a socket.
{quote}
Thanks [~fabbri] for the explanation.  In {{StandardSocketOptions}}, an attempt to set the
socket send buffer to larger than its maximum size causes it to be set to its maximum size
(refer to [link here|https://docs.oracle.com/javase/7/docs/api/java/net/StandardSocketOptions.html#SO_SNDBUF]).
I think the value for {{Socket}} would work in the same way for socket. This way, a larger
value like 512KB would not cause problems for set operation. However, the test may still fail
as the maximum size, as the default size, is system dependent.

That being said, testing the specified value is indeed tricky. To make the test assertion
covers this case, how about comparing the hinted buffer size? I see very limited chance that
the test fails, but it does confirm that, the config key works as expected.

Sample code as following:
{code:java}
  @Test
  public void testSpecifiedSendBufferSize() throws IOException {
    conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 256 * 1024);
    final Socket s1 = createSocket();
    conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 1024);
    final Socket s2 = createSocket();
    LOG.info("The real big buffer size is {}, small buffer size is {}",
        s1.getSendBufferSize(), s2.getSendBufferSize());
    assertTrue(s1.getSendBufferSize() > s2.getSendBufferSize());
}
{code}

What's your thought?

> TestDFSClientSocketSize buffer size tests are flaky
> ---------------------------------------------------
>
>                 Key: HADOOP-13351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13351
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.8.0, 3.0.0-alpha1
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13551.001.patch
>
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was set via {{java.net.Socket#setSendBufferSize}}
is equal to the value subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because {{Socket#setSendBufferSize()}}[is only a
hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
 (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message