hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Gray <jl...@streamy.com>
Subject Re: Save data to HBase
Date Wed, 02 Sep 2009 02:22:04 GMT
There's no reason to use the write buffer if you are only inserting one 
row and then immediately flushing.  This code does the exact same thing 
as yours:

table1 = new HTable("MyTable");
Put p = new Put(...);
p.add(...);
table1.put(p);

That's it.  You would only use the write-buffering if you wanted to send 
multiple Puts at once.

As far as the table only having one region, yes this limits all work on 
that table to a single node.  However with 200MB of data, you are 
smaller than the default region size of 256MB.  You can lower that split 
threshold or you can manually split in the shell or using HBaseAdmin.

Not sure about that ZooKeeper warning... Creation of /hbase should 
happen in the master at boot-time, not in a client running inserts.  Do 
you have anymore detail?  That warning showed up in the client? 
Anything in the regionserver?  Can you use the shell?

JG

y_823910@tsmc.com wrote:
> Hi,
> 
> I want to put only 200MB data to a HBase table.
> Just use a single thread to put 1MB data in a time.
> So, there are 200 job in turn.
> It occured WARN zookeeper.ZooKeeperWrapper: Failed to create /hbase,
> while it run to around the 20th job.
> I found that table has just one regin.
> I wonder whether these tight job will make that region server choke.
> About the region split, I want to know when the table's region will be
> splitted.
> Is its threshold setting in hbase-default.xml?
> Following are a snippet of my code about putting my data to hbase, anything
> can improve it?
> 
>   table1 = new HTable("MyTable");
>   table1.setAutoFlush(false);
>   table1.setWriteBufferSize(1024*1024*12);
> 
>   Put p = new Put(Bytes.toBytes(rs.getString(KeyColumn)));
>   p.add(Bytes.toBytes(FamColumn),
> Bytes.toBytes("content"),Bytes.toBytes(json));
>   table1.put(p);
> 
>   table1.flushCommits();
>   table1.close();
> 
> My cluster is only 3 PC with (1G,1G,2G RAM ,2 core CPU )
> 3 region server
> 3 ZK quorum server
> We'll enlarge our cluster to 12PCs with 8G RAM.
> 
> Thanks
> 
> Fleming
> 
> 
> 
>                                                                                     
                                                                 
>                       Amandeep Khurana                                              
                                                                 
>                       <amansk@gmail.com        To:      hbase-user@hadoop.apache.org
                                                                 
>                       >                        cc:      (bcc: Y_823910/TSMC)     
                                                                    
>                                                Subject: Re: Save data to HBase      
                                                                 
>                       2009/09/02 03:07                                              
                                                                 
>                       AM                                                            
                                                                 
>                       Please respond to                                             
                                                                 
>                       hbase-user                                                    
                                                                 
>                                                                                     
                                                                 
>                                                                                     
                                                                 
> 
> 
> 
> 
> What do you mean by quick way? You can use the api to put data into it
> through a standalone java program or you can use a MR job to do it..
> 
> 
> On Tue, Sep 1, 2009 at 2:48 AM, <y_823910@tsmc.com> wrote:
> 
>> Hi there,
>>
>> Any quick way that I can save my data(12G) to HBase?
>> Thanks
>>
>> Fleming
>>
> 
>>
>>
>>
> 
> 
> 
> 
> 
>  --------------------------------------------------------------------------- 
>                                                          TSMC PROPERTY       
>  This email communication (and any attachments) is proprietary information   
>  for the sole use of its                                                     
>  intended recipient. Any unauthorized review, use or distribution by anyone  
>  other than the intended                                                     
>  recipient is strictly prohibited.  If you are not the intended recipient,   
>  please notify the sender by                                                 
>  replying to this email, and then delete this email and any copies of it     
>  immediately. Thank you.                                                     
>  --------------------------------------------------------------------------- 
> 
> 
> 

Mime
View raw message