hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4570) Scan ACID problem with concurrent puts.
Date Sat, 15 Oct 2011 00:56:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128008#comment-13128008
] 

Todd Lipcon commented on HBASE-4570:
------------------------------------

Jon and I spent the afternoon with his test cases. We've found the issue - it's a nice one!

In KeyValue, we have the following code:
{code}
  public byte [] getRow() {
    if (rowCache == null) {
      int o = getRowOffset();
      short l = getRowLength();
      rowCache = new byte[l];
      System.arraycopy(getBuffer(), o, rowCache, 0, l);
    }
    return rowCache;
  }
{code}
which is called extensively by KeyValueHeaps throughout the scanner code. In the case of scanning
MemStore, an individual KeyValue ends up as {{next}} in multiple MemStoreScanners. Then, if
multiple threads call {{getRow}} at the same time, we see the following race:
- Thread 1 sees {{rowCache}} as null, and initializes {{rowCache = new byte[...]}}
- Thread 2 sees {{rowCache}} as non-null, and returns a byte array of all 0s
- Thread 1 initializes the row with {{arrayCopy}}, and returns the right result

The byte array returned to Thread 2 is modified while it's working with it, so depending on
the interleaving of events, it can cause an invalid heap, or invalid results, or a weird split
row like Jon was seeing, etc.

The fix is pretty simple - we need to declare {{rowCache}} volatile, and initialize it in
a temporary variable before overwriting the volatile reference. If this is too slow, we could
use an AtomicFieldUpdater with {{lazySet}} to put the cost only on the write side, but I don't
think it really matters.

                
> Scan ACID problem with concurrent puts.
> ---------------------------------------
>
>                 Key: HBASE-4570
>                 URL: https://issues.apache.org/jira/browse/HBASE-4570
>             Project: HBase
>          Issue Type: Bug
>          Components: client, regionserver
>    Affects Versions: 0.90.1, 0.90.3
>            Reporter: Jonathan Hsieh
>         Attachments: 4570-instrumentation.tgz, hbase-4570.tgz
>
>
> When scanning a table sometimes rows that have multiple column families get split into
two rows if there are concurrent writes.  In this particular case we are overwriting the contents
of a Get directly back onto itself as a Put.
> For example, this is a two cf row (with "f1", "f2", .. "f9" cfs).  It is actually returned
as two rows (#55 and #56). Interestingly if the two were merged we would have a single proper
row.
> Row row0000024461 had time stamps: [55: keyvalues={row0000024461/f0:data/1318200440867/Put/vlen=1000,
row0000024461/f0:qual/1318200440867/Put/vlen=10, row0000024461/f1:data/1318200440867/Put/vlen=1000,
row0000024461/f1:qual/1318200440867/Put/vlen=10, row0000024461/f2:data/1318200440867/Put/vlen=1000,
row0000024461/f2:qual/1318200440867/Put/vlen=10, row0000024461/f3:data/1318200440867/Put/vlen=1000,
row0000024461/f3:qual/1318200440867/Put/vlen=10, row0000024461/f4:data/1318200440867/Put/vlen=1000,
row0000024461/f4:qual/1318200440867/Put/vlen=10}, 
> 56: keyvalues={row0000024461/f5:data/1318200440867/Put/vlen=1000, row0000024461/f5:qual/1318200440867/Put/vlen=10,
row0000024461/f6:data/1318200440867/Put/vlen=1000, row0000024461/f6:qual/1318200440867/Put/vlen=10,
row0000024461/f7:data/1318200440867/Put/vlen=1000, row0000024461/f7:qual/1318200440867/Put/vlen=10,
row0000024461/f8:data/1318200440867/Put/vlen=1000, row0000024461/f8:qual/1318200440867/Put/vlen=10,
row0000024461/f9:data/1318200440867/Put/vlen=1000, row0000024461/f9:qual/1318200440867/Put/vlen=10}]
> I've only tested this on 0.90.1+patches and 0.90.3+patches, but it is consistent and
duplicatable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message