hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-576) Investigate IPC performance
Date Sat, 04 Oct 2008 06:45:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636791#action_12636791
] 

stack commented on HBASE-576:
-----------------------------

Above comment 'stack - 02/Oct/08 10:48 PM' is incorrect.  In fact it's a Client per socketfactory
rather than a Client per remote host.   A Client goes against many hosts.  For each, Client
keeps a cache of Connections keyed by remote address (and then some).  An invocation on the
remote host sends the request then sleeps till the response comes back.  During sending and
receipt of response, the connection is devoted but otherwise, is available.  Unless the request
or response large, socket is being 'multiplexed' (if large, need to chunk request/response).

Here's sample from a log with the RPC logging enabled plus some logging I added.  The log
is of 4 threads running in single JVM each running PerformanceEvaluation test (need above
patch applied).  The lone number after the name of the class is the thread name.   Threads
were named 0, 1, 2, and 3.  The numbers at the end with the '####' in front are request ids.
 See how they are interlaced.

{code}
...
08/10/03 23:16:03 INFO ipc.HBaseClient: 3 org.apache.hadoop.ipc.HBaseClient@da9ea4 start /208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user sending #####1330
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user got value #####1327
08/10/03 23:16:03 DEBUG ipc.Client: 0 FINISHED WAITING ON ###1327
08/10/03 23:16:03 INFO ipc.HBaseClient: 0 org.apache.hadoop.ipc.HBaseClient@da9ea4 done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 80
08/10/03 23:16:03 INFO ipc.HBaseClient: 0 org.apache.hadoop.ipc.HBaseClient@da9ea4 start /208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user sending #####1331
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user got value #####1328
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user got value #####1329
08/10/03 23:16:03 DEBUG ipc.Client: 1 FINISHED WAITING ON ###1328
08/10/03 23:16:03 INFO ipc.HBaseClient: 1 org.apache.hadoop.ipc.HBaseClient@da9ea4 done
08/10/03 23:16:03 DEBUG ipc.Client: 2 FINISHED WAITING ON ###1329
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 97
08/10/03 23:16:03 INFO ipc.HBaseClient: 2 org.apache.hadoop.ipc.HBaseClient@da9ea4 done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 98
08/10/03 23:16:03 INFO ipc.HBaseClient: 1 org.apache.hadoop.ipc.HBaseClient@da9ea4 start /208.76.44.139:60020
08/10/03 23:16:03 INFO ipc.HBaseClient: 2 org.apache.hadoop.ipc.HBaseClient@da9ea4 start /208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user sending #####1332
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user sending #####1333
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user got value #####1330
08/10/03 23:16:03 DEBUG ipc.Client: 3 FINISHED WAITING ON ###1330
08/10/03 23:16:03 INFO ipc.HBaseClient: 3 org.apache.hadoop.ipc.HBaseClient@da9ea4 done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 99
08/10/03 23:16:03 INFO ipc.HBaseClient: 3 org.apache.hadoop.ipc.HBaseClient@da9ea4 start /208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user sending #####1334
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user got value #####1331
08/10/03 23:16:03 DEBUG ipc.Client: 0 FINISHED WAITING ON ###1331
08/10/03 23:16:03 INFO ipc.HBaseClient: 0 org.apache.hadoop.ipc.HBaseClient@da9ea4 done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 113
08/10/03 23:16:03 INFO ipc.HBaseClient: 0 org.apache.hadoop.ipc.HBaseClient@da9ea4 start /208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user sending #####1335
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user got value #####1332
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user got value #####1333
08/10/03 23:16:03 DEBUG ipc.Client: 1 FINISHED WAITING ON ###1332
08/10/03 23:16:03 INFO ipc.HBaseClient: 1 org.apache.hadoop.ipc.HBaseClient@da9ea4 done
08/10/03 23:16:03 DEBUG ipc.Client: 2 FINISHED WAITING ON ###1333
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 178
08/10/03 23:16:03 INFO ipc.HBaseClient: 2 org.apache.hadoop.ipc.HBaseClient@da9ea4 done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 178
08/10/03 23:16:03 INFO ipc.HBaseClient: 1 org.apache.hadoop.ipc.HBaseClient@da9ea4 start /208.76.44.139:60020
08/10/03 23:16:03 INFO ipc.HBaseClient: 2 org.apache.hadoop.ipc.HBaseClient@da9ea4 start /208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user sending #####1336
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user sending #####1337
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to /208.76.44.139:60020 from
an unknown user got value #####1334
08/10/03 23:16:03 DEBUG ipc.Client: 3 FINISHED WAITING ON ###1334
08/10/03 23:16:03 INFO ipc.HBaseClient: 3 org.apache.hadoop.ipc.HBaseClient@da9ea4 done
...
{code}

Hadoop RPC doesn't seem to have a 'big lock' at its core, not unless the request or response
large. Its pretty concurrent.

Will look now at trying to hack in grizzly in the least intrusive manner just to see if grizzly
is faster.

> Investigate IPC performance
> ---------------------------
>
>                 Key: HBASE-576
>                 URL: https://issues.apache.org/jira/browse/HBASE-576
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.1.0, 0.1.1, 0.1.2, 0.2.0
>            Reporter: Jim Kellerman
>            Assignee: stack
>         Attachments: pe.patch
>
>
> Turning off all file I/O, and running the PerformanceEvaluation test, of 1,048,576 sequential
writes to HBase managed to achieve only 7,285 IPCs per second.
> Running PerformanceEvaluation sequential write test modified to do an abort instead of
a commit, it was possible to do 68,337 operations per second. We are obviously spending a
lot of time doing IPCs. 
> We need to investigate to find the bottleneck. Marshalling and unmarshalling? Socket
setup and teardown?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message