Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of jlist@streamy.com designates
 72.34.249.3 as permitted sender)
Message-ID: <55402.173.51.255.165.1251868637.squirrel@webmail.streamy.com>
In-Reply-To: 
     <OFA183C9B2.368AABC8-ON48257625.0013ED40-48257625.0013F50F@tsmc.com>
References: 
    <OFA183C9B2.368AABC8-ON48257625.0013ED40-48257625.0013F50F@tsmc.com>
Date: Tue, 1 Sep 2009 22:17:17 -0700 (PDT)
Subject: Re: Save data to HBase
From: "Jonathan Gray" <jlist@streamy.com>
To: hbase-user@hadoop.apache.org
Reply-To: jlist@streamy.com
User-Agent: SquirrelMail/1.5.1
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit

Once the JVM consumes memory it generally will not release it back to the
OS (so you will see it near-full).

How much heap are you giving the RS?

The issue you're having seems rather odd... Nothing at all changes between
the different insert runs?  Instantiating a new HTable should not be
creating a /hbase node.

On Tue, September 1, 2009 8:37 pm, y_823910@tsmc.com wrote:
> Hi,
>
>
> I guess Creation of /hbase happen after running table1 = new
> HTable("MyTable");
> Right, that warning showed up in the client.There doesn't keep any log in
> that region server. When ever I finish to put my data, I would count it by
> shell. I put the first 10 files(10M) in five threads. It worked pretty
> well. After taking a minutes break, I put next 30 files.It still worked
> very quickly. When I count it, I found it became very very slow.
> By checking that region server,I found that region server taken so much
> RAM(almost on the top),
> and seems not to release it. Maybe I should lower the flush threshold.(how
> to do it ? ^_^) If I don't take a break and continue to put data, that
> warning raised in the client. So, I wonder that region server is too busy
> to serve the following client request. If a client can't reach that busy
> sever, it would be better to raise some exception that I can catch and try
> to handle it. Thanks
>
>
> Fleming
>
>
>
>
>
>
> Jonathan Gray
> <jlist@streamy.co        To:      hbase-user@hadoop.apache.org
> m>                       cc:      (bcc: Y_823910/TSMC) Subject: Re: Save
> data to HBase 2009/09/02 10:22
> AM
> Please respond to
> hbase-user
>
>
>
>
>
>
> There's no reason to use the write buffer if you are only inserting one
> row and then immediately flushing.  This code does the exact same thing as
> yours:
>
>
> table1 = new HTable("MyTable"); Put p = new Put(...);
> p.add(...); table1.put(p);
>
> That's it.  You would only use the write-buffering if you wanted to send
> multiple Puts at once.
>
> As far as the table only having one region, yes this limits all work on
> that table to a single node.  However with 200MB of data, you are smaller
> than the default region size of 256MB.  You can lower that split threshold
> or you can manually split in the shell or using HBaseAdmin.
>
> Not sure about that ZooKeeper warning... Creation of /hbase should
> happen in the master at boot-time, not in a client running inserts.  Do you
> have anymore detail?  That warning showed up in the client? Anything in
> the regionserver?  Can you use the shell?
>
> JG
>
>
> y_823910@tsmc.com wrote:
>> Hi,
>>
>>
>> I want to put only 200MB data to a HBase table.
>> Just use a single thread to put 1MB data in a time.
>> So, there are 200 job in turn.
>> It occured WARN zookeeper.ZooKeeperWrapper: Failed to create /hbase,
>> while it run to around the 20th job. I found that table has just one
>> regin. I wonder whether these tight job will make that region server
>> choke. About the region split, I want to know when the table's region
>> will be splitted. Is its threshold setting in hbase-default.xml?
>> Following are a snippet of my code about putting my data to hbase,
>>
> anything
>> can improve it?
>>
>> table1 = new HTable("MyTable"); table1.setAutoFlush(false);
>> table1.setWriteBufferSize(1024*1024*12);
>>
>> Put p = new Put(Bytes.toBytes(rs.getString(KeyColumn)));
>> p.add(Bytes.toBytes(FamColumn),
>> Bytes.toBytes("content"),Bytes.toBytes(json));
>> table1.put(p);
>>
>> table1.flushCommits(); table1.close();
>>
>> My cluster is only 3 PC with (1G,1G,2G RAM ,2 core CPU )
>> 3 region server
>> 3 ZK quorum server
>> We'll enlarge our cluster to 12PCs with 8G RAM.
>>
>>
>> Thanks
>>
>>
>> Fleming
>>
>>
>>
>>
>>
>
>> Amandeep Khurana
>>
>
>> <amansk@gmail.com        To:
>>
> hbase-user@hadoop.apache.org
>
>>> cc:      (bcc:
>>>
> Y_823910/TSMC)
>
>
>> Subject: Re: Save data to
>>
> HBase
>
>
>> 2009/09/02 03:07
>>
>
>> AM
>>
>
>> Please respond to
>>
>
>> hbase-user
>
>>
>
>>
>
>>
>>
>>
>>
>> What do you mean by quick way? You can use the api to put data into it
>> through a standalone java program or you can use a MR job to do it..
>>
>>
>> On Tue, Sep 1, 2009 at 2:48 AM, <y_823910@tsmc.com> wrote:
>>
>>
>>> Hi there,
>>>
>>>
>>> Any quick way that I can save my data(12G) to HBase?
>>> Thanks
>>>
>>>
>>> Fleming
>>>
>>>
>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
> -------------------------------------------------------------------------
> --
>
>
>> TSMC PROPERTY
>>
>
>> This email communication (and any attachments) is proprietary
>>
> information
>> for the sole use of its
>
>> intended recipient. Any unauthorized review, use or distribution by
> anyone
>> other than the intended
>
>> recipient is strictly prohibited.  If you are not the intended
> recipient,
>> please notify the sender by
>
>> replying to this email, and then delete this email and any copies of it
>>
>
>> immediately. Thank you.
>
>>
> -------------------------------------------------------------------------
> --
>
>
>>
>>
>>
>
>
>
>
> -------------------------------------------------------------------------
> --
> TSMC PROPERTY
> This email communication (and any attachments) is proprietary information
> for the sole use of its intended recipient. Any unauthorized review, use or
> distribution by anyone other than the intended recipient is strictly
> prohibited.  If you are not the intended recipient, please notify the
> sender by replying to this email, and then delete this email and any
> copies of it immediately. Thank you.
> -------------------------------------------------------------------------
> --
>
>
>
>
>