hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyfar <dly...@gmail.com>
Subject Re: Problems with write performance (25kb rows)
Date Tue, 05 Jan 2010 13:56:12 GMT

I did some tests with different flushing parameters. I've touched following

When I've increased flushsize to 256Mb (64Mb by default) time of 25Kb test
When I've changed block.multiplier to 12 (was 10) I won about 40 seconds in
25Kb test (was 150-160 secs, became ~110 secs for one running instance that
inserts 100K records).
All tests I did was with WAL on.

2010/1/5 Dmitriy Lyfar <dlyfar@gmail.com>

> Hello Stack,
>> > And throughput without WAL is about 50 Mb/sec and  about 15 Mb/sec with
>> WAL
>> > on. When I run clients in serial order (i.e. at the moment there is only
>> > one
>> > working script) time almost stable and not grows.
>> >
>> >
>> > > See what the
>> > > numbers are like uploading into a table that is pre-split?
>> >
>> >
>> > Sorry, what you mean pre-split? You mean splitting regions before
>> running
>> > script?
>> >
>> >
>> I was thinking you were uploading into a new table and that the region
>> splits were happening inline with your upload.  I was asking what the
>> performance was like if the table had already had all its regions pre-made
>> wondering if it ran faster but sounds like your table is already
>> pre-split.
>> So where are we at now?  You tried running multiple separate upload
>> processes and it still runs too slow?
> Yes, still too slow, especially with WAL on. Btw, I see the greater row
> size, the greater impact has WAL. I'm not an expert in hbase internals, but
> I begin think that the reason of throughput fall in case of 25Kb size
> connected with flushing. I mean looks like we begin flush too often and it
> impacts on throughput.
> Also as I see from architecture description there are could be several
> reasons, like rolling hlog too often and long compaction period. Would you
> advice which log messages in region/master logs should warn me that
> something going wrong?
> --
> Regards, Lyfar Dmitriy

Regards, Lyfar Dmitriy
mailto: dlyfar@crystalnix.com
jabber: dlyfar@gmail.com

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message