accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-1053) continuous ingest detected data loss
Date Tue, 12 Feb 2013 14:41:13 GMT


Eric Newton commented on ACCUMULO-1053:

Here's a basic analysis of the data loss.  It is incomplete, but I want to document the approach.

First, I look at the reported missing entries.  They look like this:

7001d8d9c37ff1a0        259018854a9e0ab5
700255fb34764bf0        598c79d47356f8da
70031a5ad5ff67c0        017b55debd8b9ba7

You read this as "row id 7001d8d9c37ff1a0 is missing, it is referenced in row 259018854a9e0ab5".

A basic check with the shell confirms this:

root@test ci> scan -r 7001d8d9c37ff1a0
root@test ci> scan -r 259018854a9e0ab5 -st
259018854a9e0ab5 5705:3563 [] 1360613984602    74430138-9c16-4135-8230-53b2c7e9af94:000000001987ebe3:7001d8d9c37ff1a0:87abf256

The reference row contains:
||Key Element||Data||
|row| random id|
|cf:cq| random column|
|ts| ingest time|
|value| ingester uuid: counter: reference: checksum

Find the ingester by greping through the ingest logs for the ingester uuid.

>From there, find the moment when the flush occurred to look for errors:

FLUSH 1360613974304 3085438 3081429 427999984 1000000
FLUSH 1360614028073 53769 49119 428999984 1000000

Verify the count matches the timeframe (000000001987ebe3 == 428338147):

FLUSH 1360613974304 3085438 3081429 427999984 1000000
      1360613984602                 428338147 
FLUSH 1360614028073 53769 49119     428999984 1000000

It's good to verify these little assumptions because its very easy to make a small mistake
and look at the wrong log file, or misread a digit.

Find the tablet name by scanning the !METADATA table:

scan -b 5;7001d8d9c37ff1a0 -c ~tab:~pr,file
5;7020c49ba5e354a file:/t-0008a8t/A0008ld2.rf []    95094250,2306420
5;7020c49ba5e354a file:/t-0008a8t/C000919w.rf []    95750715,2319223
5;7020c49ba5e354a file:/t-0008a8t/F000921z.rf []    6838391,170103
5;7020c49ba5e354a file:/t-0008a8t/F000924e.rf []    5517442,136722
5;7020c49ba5e354a ~tab:~pr []    \x0170000000000000a8

Find the location of the tablet at the time of the loss by looking for "5;7020c49ba5e354a"
in all the tserver logs, sorted by time.

Now you can find the walogs in use by the tablet server, the updates made by the ingester
(by ip address).

In this case, the data was written to a walog, but the tablet was successfully minor compacted.
 The minor compaction file was compacted into a 2nd file.  That file was compacted into C000919w.rf.
 However C000919w.rf does not contain 7001d8d9c37ff1a0.

However, there was no read of the WAL, no recovery took place all of this took place on a
single server.

I've turned on a 48-hour trash, and will re-run the test, so I can narrow down where the data
is lost.

> continuous ingest detected data loss
> ------------------------------------
>                 Key: ACCUMULO-1053
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: test, tserver
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>            Priority: Critical
>             Fix For: 1.5.0
> Now that we're logging directly HDFS, we added datanodes to the agitator. That is, we
are now killing data nodes during ingest, and now we are losing data.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message