accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Fix "Table x has a hole" [SEC=UNOFFICIAL]
Date Tue, 28 Feb 2017 16:36:19 GMT
Below are some commands that show how to recreate this problem and how
to fix it.   Each table in the metadata table has a pointer to the
previous tablets.  Adding and removing splits to a table changes this.

  root@uno> createtable test

Get the tables ID below we will need it later.

  root@uno test> tables -l
  accumulo.metadata    =>        !0
  accumulo.replication =>      +rep
  accumulo.root        =>        +r
  test                 =>         3
  trace                =>         1

Add some splits and then scan the metadata table.  The pointers to the
previous tablet are in the ~tab:~pr column.  The scan below uses the
table id above.

  root@uno test> addsplits 11111111 3333333
  root@uno test> scan -t accumulo.metadata -c ~tab:~pr -b 3; -e 3<
  3;11111111 ~tab:~pr []    \x00
  3;3333333 ~tab:~pr []    \x0111111111
  3< ~tab:~pr []    \x013333333

Add another split and rescan the metadata table.

  root@uno test> addsplits 2222222
  root@uno test> scan -t accumulo.metadata -c ~tab:~pr -b 3; -e 3<
  3;11111111 ~tab:~pr []    \x00
  3;2222222 ~tab:~pr []    \x0111111111
  3;3333333 ~tab:~pr []    \x012222222
  3< ~tab:~pr []    \x013333333

Grant permission to write to the metadata table and then recreate the
problem you have.

  root@uno test> grant Table.WRITE -u root -t accumulo.metadata
  root@uno test> table accumulo.metadata
  root@uno accumulo.metadata> insert 3;3333333 ~tab ~pr \x0111111111
  root@uno accumulo.metadata> scan -t accumulo.metadata -c ~tab:~pr -b 3; -e 3<
  3;11111111 ~tab:~pr []    \x00
  3;2222222 ~tab:~pr []    \x0111111111
  3;3333333 ~tab:~pr []    \x0111111111
  3< ~tab:~pr []    \x013333333

If you ran check for metadata problems here, should see the error
message you saw.  Below, the pointer is fixed and write permission is
revoked (to prevent accidental writes in the future).

  root@uno accumulo.metadata> insert 3;3333333 ~tab ~pr \x012222222
  root@uno accumulo.metadata> revoke Table.WRITE -u root -t accumulo.metadata
  root@uno accumulo.metadata>

After running the command above to fix the potiner, check for metadata
problems should be happy.

It would be nice to try to track down the cause of this.  Spliting a
tablet involves three metadata operations.  For fault tolerance, the
columns ~tab:oldprevrow and ~tab:splitRatio are temporarily written.
If a tablet server dies in the middle of splitting a tablet, then
Accumulo will see these temporary columns and attempt to continue the
split.  So I am curious if you see these columns?

On Sun, Feb 26, 2017 at 6:49 PM, Dickson, Matt MR
<matt.dickson@defence.gov.au> wrote:
> UNOFFICIAL
>
> Running the CheckForMetadataProblems on Accumulo is listing
>
> Table xxx has a hole 11111111 != 2222222
>
> Is there a correct way to repair this?
>
> Thanks in advance.

Mime
View raw message