hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Server-side write buffer configuration
Date Wed, 11 Aug 2010 16:40:34 GMT
By multiple JVMs I mean multiple HBase clients, it's hard to get more
basic than that. Were you planning on doing that 100TB upload from a
single client? If so, you should revise your plans. Do split the input
between many inserting processes, use MapReduce if you can as it will
do that for you, and you will leverage the parallelism offered by
HBase/Hadoop.

For the rest, remember that google is your friend.

J-D

On Tue, Aug 10, 2010 at 10:42 PM, Han Liu <hanl1@andrew.cmu.edu> wrote:
> Hi J-D,
>
> Can you explain a bit more about multiple JVMs´╝č For example how to use it in the case
of HBase clients? Or maybe point me to a reference on such topics since I am not really an
expert on Java. :p
>
> Thanks again for your reply.
>
> Han
>
>
> On Aug 9, 2010, at 4:13 PM, Jean-Daniel Cryans wrote:
>
>>> I see. 0.89 is the still a developer release and I hear that it is not stable.
But it sounds really tempting because it boosts performance by a lot. Can I trust it if my
final goal is to insert about 100TB of data? What could be the possible issues? Also when
shall I expect to see a stable release?
>>
>> The problem will be your 6 machines, not the software. And if you need
>> to insert that much data, please use the bulk uploader as it will be
>> much faster: http://hbase.apache.org/docs/r0.89.20100621/bulk-loads.html
>>
>>>
>>> So in order to do multiple-client insertion I basically just need to create multiple
HTable objects to handle the insertions? Do I need to do multi-threading manually?
>>
>> By multiple clients I mean multiple JVMs, you can also do
>> multi-threading inside each client with their own HTable (since it's
>> not thread-safe). But please use the bulk loader.
>>
>>>
>>> And finally a possibly stupid question: how do I check what is the total number
of regions?
>>
>> Check the "status" command in the shell, or checkout the master web UI
>> on port 60030 like it says here
>> http://hbase.apache.org/docs/r0.20.6/api/overview-summary.html#runandconfirm
>>
>> J-D
>>
>
> --
> Han Liu
> SCS & HCI Institute
> Undergrad. Class of 2012
> Carnegie Mellon University
>
>
>
>
>

Mime
View raw message