Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of ruslan.usifov@gmail.com
 designates 209.85.220.224 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=JXWTB9/0D7dUKPhlcZRHggLg5bAoWdOZMTozep4VL81grPDARHxmJb5wFi6ru/rcvB
         UG62Sp3/5Hs8bfxeRIlSpn4z0Av7CJsUDFZeEFRrAVHc7lavt2ygp0rAP1JAl6Bv8gcg
         BLo3LEeQsZUM7PInTMSgKX+lBO9T/0IT+O6K4=
MIME-Version: 1.0
In-Reply-To: <9b048d180912080703s21ea880ame1bbdbc70d6eaf65@mail.gmail.com>
References: <9b048d180912080703s21ea880ame1bbdbc70d6eaf65@mail.gmail.com>
Date: Tue, 8 Dec 2009 23:54:45 +0300
Message-ID: <9b048d180912081254l40394881n5ccd71b5fc6375a4@mail.gmail.com>
Subject: Concurrent reads throw thrift
From: ruslan usifov <ruslan.usifov@gmail.com>
To: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023545bd658b8db95047a3dc80b

--0023545bd658b8db95047a3dc80b
Content-Type: text/plain; charset=ISO-8859-1

Hi!

I have cluster from 3 machines:

firts(srv1) have NameNode, DataNode, Hmaster, Hregionserver, Thrift on it
second(srv2) have DataNode, Hregionserver, HQuorumPerr, Thrift on it.
third(srv3) have DataNode, Hregionserver

and have follow python program, that run in 8 separate process

import random;
import time;

from thrift import Thrift
from thrift.transport import TSocket, TTransport
from thrift.protocol import TBinaryProtocol
from hbase import ttypes
from hbase.Hbase import Client, ColumnDescriptor, Mutation, BatchMutation

def get_client(host):
  transport = TSocket.TSocket(host, 9090)

  # Buffering is critical. Raw sockets are very slow
  transport = TTransport.TBufferedTransport(transport)

  # Wrap in a protocol
  protocol = TBinaryProtocol.TBinaryProtocolAccelerated(transport)

  # Create a client to use the protocol encoder
  client = Client(protocol)

  # Connect!
  transport.open()

  return client

clients = [get_client("192.168.0.241"), get_client("192.168.0.242")];
begin = time.time();
maxrange = 20000000 - 1;

for i in xrange(500000):
  if (i != 0) and ((i % 10000) == 0):
    tm = time.time();
    print tm - begin;
    begin = tm;

  client = clients[random.randint(0, len(clients) - 1)];
  client.getRow("test", str(random.randint(0, maxrange)));

print time.time() - begin;


And i have very big latency (about 300 request per seconds). If i run that
program in 4 process latency is slow but i think is not good (making about
1000 request per second) It's very strange that and cpu and discs are not
100% load
Please help!!! What i do wrong?

--0023545bd658b8db95047a3dc80b--