hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
Date Tue, 20 Mar 2012 21:55:39 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233845#comment-13233845

Lars Hofhansl commented on HBASE-5569:

So HBASE-2856 has logic that prevents KVs from being removed in a flush or compaction when
it has expired (due to TTL or too many version) but there is still a scanner open with a readpoint
<= the KV's memstoreTS. (which means these KVs were created after the scanner was opened)
Say you have set your store set 3 versions. Now you create 10 versions of a KV, the extra
7 versions are not removed during a flush or compaction when a scanner that was opened before
the KVs were created.
This patch adds the same for deleted KVs (IMHO that is something that HBASE-2856 missed).
So now expired and deleted KVs are not collected if a scanner could still access them.

It means that a flush or compaction needs to copy these KVs to the new store file instead
of skipping them. This only happens for KVs that were created (or now deleted) after the scanner(s)
were openened.

The output I added is the memstoreTS. The ts is already part toString.

I don't think that needs to be part of the release notes (at least we did not add this to
the release notes for HBASE-2856 or its backport).

> Do not collect deleted KVs when they are still in use by a scanner.
> -------------------------------------------------------------------
>                 Key: HBASE-5569
>                 URL: https://issues.apache.org/jira/browse/HBASE-5569
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Lars Hofhansl
>             Fix For: 0.94.0, 0.96.0
>         Attachments: 5569-v2.txt, 5569-v3.txt, 5569-v4.txt, 5569.txt, TestAtomicOperation-output.trunk_120313.rar
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected when in use
by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499): Storescanner.peek()
is changed where before = rowB/colfamily11:qual1/75366/Put/vlen=6,and after = rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the Put the thread
see 0 KVs, if the "before" was the Delete the threads see 2 KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still some consistency
issue with scanning sometimes :(

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message