From user-return-55790-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Thu Mar 21 01:34:27 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id A5BED18062C for ; Thu, 21 Mar 2019 02:34:26 +0100 (CET) Received: (qmail 15727 invoked by uid 500); 21 Mar 2019 01:34:25 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 15716 invoked by uid 99); 21 Mar 2019 01:34:25 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Mar 2019 01:34:25 +0000 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 7DB419494 for ; Thu, 21 Mar 2019 01:34:24 +0000 (UTC) Received: by mail-qt1-f179.google.com with SMTP id z17so3923051qts.13 for ; Wed, 20 Mar 2019 18:34:24 -0700 (PDT) X-Gm-Message-State: APjAAAX2h/IpxeqKitdgUs6naEQ2Kr3NkyGnI6TzhrcR6KHrKSI0jf+q UkmOTQCDjO99updjgDCAdi255f/8Ub7+dGMtzf8= X-Google-Smtp-Source: APXvYqy+tksY+qE7n7MXoaRwyVCsvVAHGJQz2U0h/w9F8bjbgq2/VPDhsYuvU93ksDyhiQEoG4ykfc4I1C9lW/ZZ9wI= X-Received: by 2002:ac8:32f2:: with SMTP id a47mr818317qtb.251.1553132064191; Wed, 20 Mar 2019 18:34:24 -0700 (PDT) MIME-Version: 1.0 References: <15d97b615c547c6fa8cb34e6d30838bf@ccri.com> In-Reply-To: <15d97b615c547c6fa8cb34e6d30838bf@ccri.com> From: Sean Busbey Date: Wed, 20 Mar 2019 20:34:12 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Bits getting flipped in record value To: aheyne Cc: user@hbase.apache.org Content-Type: multipart/alternative; boundary="000000000000f9bb1c058490bc14" --000000000000f9bb1c058490bc14 Content-Type: text/plain; charset="UTF-8" So you're saying no records should ever be updated, right? Do you have any coprocessors loaded? On Wed, Mar 20, 2019, 20:32 aheyne wrote: > I don't have the WALs but due to the nature of the data each record/key > is unique. The keys for the data are generated using spatial-temporal > dimensions of the observation. > > -Austin > > On 2019-03-20 21:25, Sean Busbey wrote: > > Have you examined the wals for writes to the impacted cells to verify > > an > > update wasn't written with the change to the value? > > > > On Wed, Mar 20, 2019, 17:47 Austin Heyne wrote: > > > >> Hey all, > >> > >> We're running HBase 1.4.8 on EMR 5.20 backed by S3 and we're seeing a > >> bit get flipped in some record values. > >> > >> We've preformed a bulk ingest and bulk load of a large chunk of data > >> and > >> then pointed a live ingest feed to that table. After a period of time > >> we > >> found that a few records in the table had been corrupted and were one > >> bit different from their original value. Since we saved the output of > >> the bulk ingest we re-loaded those files and verified that at the time > >> of bulk load the record was correct. This seems to us to indicate that > >> at some point during the live ingest writes the record was corrupted. > >> > >> I've verified that the region that the record is in has never been > >> split > >> but it has received over 2 million write requests so there very likely > >> could have been some minor compactions there. > >> > >> Has anyone seen anything like this before? > >> > >> Thanks, > >> Austin > >> > >> -- > >> Austin L. Heyne > >> > >> > --000000000000f9bb1c058490bc14--