hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Iulia Zidaru <iulia.zid...@1and1.ro>
Subject Re: Performance degradation when deleting columns
Date Wed, 09 Mar 2011 09:33:00 GMT
  Thank you very much J-D. Your replay is very useful because we are 
working to change the delete scenario and we have to understand what 
happens inside HBase because it's impossible to entirely change deletes 
with put operations.
The numbers are small because I did some tests on my local machine. On 
our testing cluster we have much higher values, but the performance 
degrade is still present. And, yes, we are hitting the same region over 
and over again.

In my understanding things happen like this:

Test 1(delete column1)
- load in MemStore the latest version for the row
- mark column 1 as deleted(insert a tombstone in MemStore)

Test 2(delete column2)
- load in MemStore the latest version for the row
- mark column 1 and column 2 as deleted(insert a tombstone in MemStore)


Every scan should also avoid columns marked as deleted, so it has more 
and more columns to avoid. Is this true?

What is not entirely clear is what to do to have a good scan performance 
again.
I see here http://outerthought.org/blog/465-ot.html that there are some 
operations on regions: flush, compaction and major compaction.
Deletes in the file system are performed only on a major compaction.
In which way the deleted rows are loaded into MemStore after a flush or 
a minor compaction. Are they loaded with a tombstone or they are not 
loaded at all?

Thank you,
Iulia




On 03/08/2011 08:35 PM, Jean-Daniel Cryans wrote:
> That's a weird use case Iulia ;)
>
> So those numbers are pretty small, for all I know you could be hitting
> the same region over and over again... so each run of the test
> influences the next one.
>
> In any case, deletes are special operations where it has the do a Get
> for each column that's specified in order to find the latest insert
> before inserting a tombstone in the MemStore. This pretty expensive
> and can generate a lot of churn in the block cache. It may be worth
> more investigation tho, Ryan might have something to add about that.
>
> A Put operation is really just an insert in the MemStore, it doesn't
> read from any file, so it's supposed to be faster than a Delete.
>
> J-D
>
> On Tue, Mar 8, 2011 at 4:30 AM, Iulia Zidaru<iulia.zidaru@1and1.ro>  wrote:
>>   Hi,
>> We have some problems when performing a large amounts of deletes.
>>
>> We are using ASF HBase 0.90 with cloudera distribution for HDFS(cdh3b3)). We
>> store in HBase the inverted index of some documents. We get a constant
>> throughput in inserting documents and in scanning the table, but we have
>> problems when some deletes are performed.
>>
>> We did some tests (10 000) with the following operations:
>> - scan some rows (a few: 2-3 rows)
>> - delete some columns for the previous scanned rows(one column per test
>> deleted)
>> - add some columns for the previous scanned rows
>>
>> We got a huge degradation in performance in both scan and deletes.
>> - scan took from 5 to 15 milliseconds (for first 10 000)
>> - delete took from 4 to 13 milliseconds
>> and it continued to decrease. After 30000 operations a scan took 45
>> milliseconds and a delete 16 milliseconds.
>>
>> The put operation was (almost) constant in time (3.99 ms to 4.5ms after
>> 30000 tests).
>>
>> On long time tests, we had the same performance degradation, but it seems
>> that at some point the performance is up again. It might be a major
>> compaction, a disk flush or what?
>>
>> In which way deletes affect the scan operation? How can we minimize these
>> effects? Do you know what operation will put the database in the optimal
>> state?
>>
>> Thank you,
>> Iulia
>>
>>
>>
>> --
>> Iulia Zidaru
>> Java Developer
>>
>> 1&1 Internet AG - Bucharest/Romania - Web Components Romania
>> 18 Mircea Eliade St
>> Sect 1, Bucharest
>> RO Bucharest, 012015
>> iulia.zidaru@1and1.ro
>> 0040 31 223 9153
>>
>>
>>


-- 
Iulia Zidaru
Java Developer

1&1 Internet AG - Bucharest/Romania - Web Components Romania
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
iulia.zidaru@1and1.ro
0040 31 223 9153

  


Mime
View raw message