Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <391236654.1223350064329.JavaMail.jira@brutus>
Date: Mon, 6 Oct 2008 20:27:44 -0700 (PDT)
From: "stack (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Commented: (HBASE-576) Investigate IPC performance
In-Reply-To: <523561413.1207948324938.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637353#action_12637353 ] 

stack commented on HBASE-576:
-----------------------------

Thanks J-D. Patch looks good.  Pity couldn't be fixed better but yeah, would need migration script.  As is will save a bunch of churn.  Let me commit it.

Looking at rpc, I see I broke it a while back; I removed the very reason we subclass RPC.  I replaced all our carefully planted HbaseObjectWritables with default ObjectWritables.  Means we're sending Strings instead of codes for our parameter names.

So, did a test where a cluster had 1M rows loaded into 11 regions spread over 3 machines.  A single client could random-read at ~482/second.  Using above patch and running with 8 threads, was able to read at 1531/second.  Basic formula: throughput can be multiplied by # of threads up to maximum of number of cluster members: e.g. if 8 threads but only 3 servers, can only see 3X throughput improvement.  If 8 servers hosting regions, should see 8X.


> Investigate IPC performance
> ---------------------------
>
>                 Key: HBASE-576
>                 URL: https://issues.apache.org/jira/browse/HBASE-576
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.1.0, 0.1.1, 0.1.2, 0.2.0
>            Reporter: Jim Kellerman
>            Assignee: stack
>         Attachments: htd.patch, pe.patch
>
>
> Turning off all file I/O, and running the PerformanceEvaluation test, of 1,048,576 sequential writes to HBase managed to achieve only 7,285 IPCs per second.
> Running PerformanceEvaluation sequential write test modified to do an abort instead of a commit, it was possible to do 68,337 operations per second. We are obviously spending a lot of time doing IPCs. 
> We need to investigate to find the bottleneck. Marshalling and unmarshalling? Socket setup and teardown?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.