Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 15AD11096F for ; Thu, 11 Jul 2013 17:53:44 +0000 (UTC) Received: (qmail 8743 invoked by uid 500); 11 Jul 2013 17:53:39 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 8665 invoked by uid 500); 11 Jul 2013 17:53:39 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 8657 invoked by uid 99); 11 Jul 2013 17:53:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Jul 2013 17:53:38 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.220.177 as permitted sender) Received: from [209.85.220.177] (HELO mail-vc0-f177.google.com) (209.85.220.177) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Jul 2013 17:53:33 +0000 Received: by mail-vc0-f177.google.com with SMTP id hv10so6875325vcb.22 for ; Thu, 11 Jul 2013 10:53:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=nVV2HlgPaRgT4dIGPCDf+lwf1I5IN+5XYS12FFO+PJ4=; b=lA6il/M+6mWnsyZrxzFgpjQB/GwsukBz2Nb/2TO4uIQ6ZLYgWWaxcPqtmNQTswDiww YZ6XFXAdUFbHx4n/sk19dxvMxr7ceJcmOvFRzBDi566zX7PC3gM2HDiGM86r7egG4lzV aZs6cJ1MK3w/NV/cExFq6ZLZiFCzOAj2xfUN4QEX8HEmgew/IJqSdUyU4AtMhSwyu+4J OyvUrP66Rev6EjjY/y4gfMbRV2tl9rx9LDuF0rgWyxKMHWBhVT8pGNM3qhiElqEVC/we mfa628o1EYShlDnBphqrWqOvlR2PnUrQgeO+CewRFuSgKBEFTZIg4FsVZBHS63dR7s/B vR6w== MIME-Version: 1.0 X-Received: by 10.52.117.77 with SMTP id kc13mr18868694vdb.67.1373565192881; Thu, 11 Jul 2013 10:53:12 -0700 (PDT) Sender: jdcryans@gmail.com Received: by 10.220.177.68 with HTTP; Thu, 11 Jul 2013 10:53:12 -0700 (PDT) In-Reply-To: References: Date: Thu, 11 Jul 2013 10:53:12 -0700 X-Google-Sender-Auth: PZpEG3h0P5chOUdSbUMUXZxCmtk Message-ID: Subject: Re: Replication - some timestamps off by 1 ms From: Jean-Daniel Cryans To: "user@hbase.apache.org" Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org Are those incremented cells? J-D On Thu, Jul 11, 2013 at 10:23 AM, Patrick Schless wrote: > I have had replication running for about a week now, and have had a lot of > data flowing to our slave cluster over that time. Now, I'm running the > verifyrep MR job over a 1-hour period a couple days ago (which should be > fully replicated), and I'm seeing a small number of "BADROWS". > Spot-checking a few of them, the issue seems to be that the rows are > present, and have the same values, but a single cell in the row will be off > by 1ms. > > For instance, the log reports this error: > java.lang.Exception: This result was different: > keyvalues={01e581745c6a43aba01adf105af4e4a92013071015/data:!\xDF\xE0\x01/1373470622986/Put/vlen=8, > 01e581745c6a43aba01adf105af4e4a92013071015/data:&s\xC0\x01/1373470923084/Put/vlen=8, > 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223717/Put/vlen=8, > 01e581745c6a43aba01adf105af4e4a92013071015/data:/\x9B\x80\x01/1373471523316/Put/vlen=8, > 01e581745c6a43aba01adf105af4e4a92013071015/data:4/`\x01/1373471822913/Put/vlen=8} > compared to > keyvalues={01e581745c6a43aba01adf105af4e4a92013071015/data:!\xDF\xE0\x01/1373470622986/Put/vlen=8, > 01e581745c6a43aba01adf105af4e4a92013071015/data:&s\xC0\x01/1373470923084/Put/vlen=8, > 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223716/Put/vlen=8, > 01e581745c6a43aba01adf105af4e4a92013071015/data:/\x9B\x80\x01/1373471523316/Put/vlen=8, > 01e581745c6a43aba01adf105af4e4a92013071015/data:4/`\x01/1373471822913/Put/vlen=8} > > Some diffing reduces the issue down to: > 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223717/Put/vlen=8 > compared to > 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223716/Put/vlen=8. > > I'm assuming that the value before "/Put" is the cell's timestamp, which > means that the copies are off by 1ms. > > Any idea what could cause this? So far (the job is still running), the > problem seems rare (about 0.05% of rows). > > Thanks, > Patrick