hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4797) RPC Server can leave a lot of direct buffers
Date Fri, 12 Dec 2008 19:19:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656122#action_12656122
] 

Raghu Angadi commented on HADOOP-4797:
--------------------------------------

Thanks Konstantin.

Both are already tested (big thanks to Koji). Not only it does not degrade performance, it
improves it (as noted in comment on Dec 8th) for large responses.  For e.g. if the response
is 10MB :
  - without this patch :
       --  it might take say 100 write operations (each time buffer size decreasing slightly)

       -- That implies JVM copies all 10MB  whopping 50 times !!.

 - with this patch :
       -- it will take 1250 write system calls (each writing 8KB) 
       -- But there is only one copy made.
       -- 8KB limit can be increased to a larger one, though not required. RPC server is not
meant for serving very large responses anyway.
       -- In my experience most people underestimate cost of buffer copies and over estimate
cost of system call.
     
Regd the tests : 
    - It was shown that if you list a directory in a loop, each time creating a new directory
there, the total virtual memory taken by NN shoots up. This patch does prevent that.
    - In the same test, NN does take less CPU. But one of the stats missing was how fast we
were able to list and create the directories. 
           -- I think actual benefit is even better the raw CPU in the test shows.


> RPC Server can leave a lot of direct buffers 
> ---------------------------------------------
>
>                 Key: HADOOP-4797
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4797
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.17.0
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>            Priority: Blocker
>             Fix For: 0.18.3, 0.19.1, 0.20.0
>
>         Attachments: HADOOP-4797-branch-18.patch, HADOOP-4797-branch-18.patch, HADOOP-4797-branch-18.patch,
HADOOP-4797.patch
>
>
> RPC server unwittingly can soft-leak direct buffers. One observed case is that one of
the namenodes at Yahoo took 40GB of virtual memory though it was configured for 24GB memory.
Most of the memory outside Java heap expected to be direct buffers. This shown to be because
of how RPC server reads and writes serialized data. The cause and proposed fix are in following
comment.
>   

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message