hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aheyne <ahe...@ccri.com>
Subject Re: Bits getting flipped in record value
Date Thu, 21 Mar 2019 01:32:38 GMT
I don't have the WALs but due to the nature of the data each record/key 
is unique. The keys for the data are generated using spatial-temporal 
dimensions of the observation.


On 2019-03-20 21:25, Sean Busbey wrote:
> Have you examined the wals for writes to the impacted cells to verify 
> an
> update wasn't written with the change to the value?
> On Wed, Mar 20, 2019, 17:47 Austin Heyne <aheyne@ccri.com> wrote:
>> Hey all,
>> We're running HBase 1.4.8 on EMR 5.20 backed by S3 and we're seeing a
>> bit get flipped in some record values.
>> We've preformed a bulk ingest and bulk load of a large chunk of data 
>> and
>> then pointed a live ingest feed to that table. After a period of time 
>> we
>> found that a few records in the table had been corrupted and were one
>> bit different from their original value. Since we saved the output of
>> the bulk ingest we re-loaded those files and verified that at the time
>> of bulk load the record was correct. This seems to us to indicate that
>> at some point during the live ingest writes the record was corrupted.
>> I've verified that the region that the record is in has never been 
>> split
>> but it has received over 2 million write requests so there very likely
>> could have been some minor compactions there.
>> Has anyone seen anything like this before?
>> Thanks,
>> Austin
>> --
>> Austin L. Heyne

View raw message