hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Marc Spaggiari (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10702) HBase fails to respect Deletes
Date Fri, 07 Mar 2014 22:35:42 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924416#comment-13924416
] 

Jean-Marc Spaggiari commented on HBASE-10702:
---------------------------------------------

As you can see, it's a very very simple test. Just deletes and gets. Not playing with the
timestamps nor the parameters nor the wal nor anything else like that. To respect the real
usecase, *1 should be replaces by * 1000 but I changed that to get the results faster.

Table is initially loaded with sequencialWrites 30K rows.

So we are talking here to about 1 500 000 puts/day and 40 000 deletes/day. Which is still
very small.

{code}

import java.util.Random;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.util.Bytes;

public class DeleteGetTest {

  public static void main(String[] args) {

    long lastPut = 0;
    long lastDelete = 0;
    long countPut = 0;
    long countDelete = 0;
    long oneDay = 3600*24; // in seconds
    long putDelay = oneDay/1500; // in seconds
    long deleteDelay = oneDay / 40;
    Random random = new Random (System.currentTimeMillis());
    long startTime = System.currentTimeMillis();

    try {
      HTable table = new HTable(HBaseConfiguration.create(),Bytes.toBytes("TestTable"));
      while (true) {
        // Check if we need to do a put.
        if (System.currentTimeMillis() > lastPut + putDelay * 1) {
          lastPut = System.currentTimeMillis();
          countPut++;
          String key = String.format("%010d", random.nextInt(100000));
          Put put = new Put(Bytes.toBytes(key));
          put.add(Bytes.toBytes("info"), Bytes.toBytes("data"), Bytes.toBytes("UUUUUUUUFFFFFFFFHHHHHHHHHHHHHHHHGGGGGGGGGGGGGGGGYYYYYYYYHHHHHHHHLLLLLLLLMMMMMMMMXXXXXXXXIIIIIIIIYYYYYYYYTTTTTTTTUUUUUUU"
+
                                                          "UUUUUUUUUPPPPPPPPGGGGGGGGLLLLLLLLZZZZZZZZZZZZZZZZOOOOOOOOLLLLLLLLKKKKKKKKMMMMMMMMOOOOOOOOHHHHHHHHRRRRRRRRPPPPPPPPPPPPPPPPEEEEEEEEWWWWWWWWLLLLLLLLRRRRRRRRVVVVVVVVKKKKKKK"
+
                                                          "KCCCCCCCCLLLLLLLLHHHHHHHHIIIIIIIIVVVVVVVVPPPPPPPPDDDDDDDDQQQQQQQQVVVVVVVVHHHHHHHHVVVVVVVVIIIIIIIIEEEEEEEEIIIIIIIIEEEEEEEEFFFFFFFFSSSSSSSSSSSSSSSSFFFFFFFFNNNNNNNNDDDDDDD"
+
                                                          "DJJJJJJJJQQQQQQQQWWWWWWWWXXXXXXXXWWWWWWWWZZZZZZZZAAAAAAAAZZZZZZZZGGGGGGGGIIIIIIIIXXXXXXXXZZZZZZZZDDDDDDDDQQQQQQQQYYYYYYYYJJJJJJJJSSSSSSSSHHHHHHHHQQQQQQQQWWWWWWWWIIIIIII"
+
                                                          "IMMMMMMMMCCCCCCCCMMMMMMMMMMMMMMMMOOOOOOOOVVVVVVVVYYYYYYYYQQQQQQQQCCCCCCCCQQQQQQQQTTTTTTTTTTTTTTTTCCCCCCCCXXXXXXXXSSSSSSSSHHHHHHHHOOOOOOOOWWWWWWWWFFFFFFFFLLLLLLLLYYYYYYY"
+
                                                          "YIIIIIIIIYYYYYYYYJJJJJJJJZZZZZZZZAAAAAAAAFFFFFFFFRRRRRRRRUUUUUUUURRRRRRRRRRRRRRRRKKKKKKKKSSSSSSSSRRRRRRRRIIIIIIIIEEEEEEEEIIIIIIIIOOOOOOOOCCCCCCCCGGGGGGGGLLLLLLLLWWWWWWW"
+
                                                          "WEEEEEEEEXXXXXXXXAAAAAAAAKKKKKKKKBBBBBBBB"));
          table.put(put);
        }
        // Check if we need to do a delete
        if (System.currentTimeMillis() > lastDelete + deleteDelay * 1) {
          lastDelete = System.currentTimeMillis();
          countDelete++;
          boolean exist = false;
          String key = null;
          Get get = null;
          while (!exist) {
            key = String.format("%010d", random.nextInt(100000));
            get = new Get(Bytes.toBytes(key));
            get.addColumn(Bytes.toBytes("info"), Bytes.toBytes("data"));
            exist = table.exists(get);
          }
          Delete delete = new Delete (Bytes.toBytes(key));
          delete.deleteColumn(Bytes.toBytes("info"), Bytes.toBytes("data"));
          table.delete(delete);
          if (!table.get(get).isEmpty()) {
            System.out.println ((System.currentTimeMillis() - startTime) + " Doing a delete
for " + key + " failed. Start to count");
            int count = 1;
            while (!table.get(get).isEmpty()) {
              table.delete(delete);
              count++;
            }
            System.out.println ("puts=" + countPut + " deletes=" + countDelete + " retries="
+ count);
            if (count > 5) {
              table.close();
              System.exit(0);
            }
          }
        }
      }
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

{code}

> HBase fails to respect Deletes
> ------------------------------
>
>                 Key: HBASE-10702
>                 URL: https://issues.apache.org/jira/browse/HBASE-10702
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.94.2, 0.94.15, 0.94.17
>            Reporter: Jean-Marc Spaggiari
>            Priority: Critical
>
> One of our user contacted me about an issue with Deletes.
> Some of the deletes they do are not totally processed. Therefore, after the Delete, if
they do a Get, from time to time, the Get return the row when it should have been deleted
and should have returned nothing. After multiple Deletes, the row is finally deleted. If we
don't retry after the 1st attempt, the row stays there. Even after a flush, a major_compact,
etc.
> I have been able to reproduce the issue in 0.94.2 (CDH4.2.0 EC2), 0.94.15(CDH4.6.0 EC2)
and 0.94.17 (Apache version bare metal)
> Here is a simple output from my test app.
> 1736509 Doing a delete for 0000099676 failed. Start to count
> puts=311 deletes=64 retries=2
> 2281712 Doing a delete for 0000027606 failed. Start to count
> puts=3679 deletes=247 retries=2
> 2388305 Doing a delete for 0000018306 failed. Start to count
> puts=4744 deletes=290 retries=2
> 2532943 Doing a delete for 0000030446 failed. Start to count
> puts=5678 deletes=337 retries=2
> 2551421 Doing a delete for 0000046304 failed. Start to count
> puts=5845 deletes=345 retries=2
> 2561099 Doing a delete for 0000019619 failed. Start to count
> puts=5869 deletes=347 retries=3
> First field is the time in ms since the test started.  So first error occurs after about
30 minutes. Below are the number of puts and deletes done, and the numbers of required retries
to get the value deleted.
> Key is random number between  0000000000 and 0000100000.
> Very simple test. Just doing more puts than deletes.
> Tests are running on 0.96.1.1 for almost 1h now so it seems to be fine, but it's not
on the same cluster, so I will keep that running for hours/days first.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message