hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyfar <dly...@gmail.com>
Subject Re: Problems with write performance (25kb rows)
Date Tue, 05 Jan 2010 13:56:12 GMT
Stack,

I did some tests with different flushing parameters. I've touched following
params:
hbase.hregion.memstore.block.multiplier
hbase.hregion.memstore.flush.size

When I've increased flushsize to 256Mb (64Mb by default) time of 25Kb test
grew.
When I've changed block.multiplier to 12 (was 10) I won about 40 seconds in
25Kb test (was 150-160 secs, became ~110 secs for one running instance that
inserts 100K records).
All tests I did was with WAL on.


2010/1/5 Dmitriy Lyfar <dlyfar@gmail.com>

> Hello Stack,
>
>
>> > And throughput without WAL is about 50 Mb/sec and  about 15 Mb/sec with
>> WAL
>> > on. When I run clients in serial order (i.e. at the moment there is only
>> > one
>> > working script) time almost stable and not grows.
>> >
>> >
>> > > See what the
>> > > numbers are like uploading into a table that is pre-split?
>> >
>> >
>> > Sorry, what you mean pre-split? You mean splitting regions before
>> running
>> > script?
>> >
>> >
>> I was thinking you were uploading into a new table and that the region
>> splits were happening inline with your upload.  I was asking what the
>> performance was like if the table had already had all its regions pre-made
>> wondering if it ran faster but sounds like your table is already
>> pre-split.
>>
>> So where are we at now?  You tried running multiple separate upload
>> processes and it still runs too slow?
>>
>
> Yes, still too slow, especially with WAL on. Btw, I see the greater row
> size, the greater impact has WAL. I'm not an expert in hbase internals, but
> I begin think that the reason of throughput fall in case of 25Kb size
> connected with flushing. I mean looks like we begin flush too often and it
> impacts on throughput.
> Also as I see from architecture description there are could be several
> reasons, like rolling hlog too often and long compaction period. Would you
> advice which log messages in region/master logs should warn me that
> something going wrong?
>
>
> --
> Regards, Lyfar Dmitriy
>
>


-- 
Regards, Lyfar Dmitriy
mailto: dlyfar@crystalnix.com
jabber: dlyfar@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message