Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 364F7200B54 for ; Thu, 14 Jul 2016 00:26:03 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 349C7160A6E; Wed, 13 Jul 2016 22:26:03 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 07653160A6A for ; Thu, 14 Jul 2016 00:26:01 +0200 (CEST) Received: (qmail 9696 invoked by uid 500); 13 Jul 2016 22:26:01 -0000 Mailing-List: contact user-help@predictionio.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@predictionio.incubator.apache.org Delivered-To: mailing list user@predictionio.incubator.apache.org Received: (qmail 9686 invoked by uid 99); 13 Jul 2016 22:26:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jul 2016 22:26:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BF6C3C0457 for ; Wed, 13 Jul 2016 22:26:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.53 X-Spam-Level: * X-Spam-Status: No, score=1.53 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=occamsmachete-com.20150623.gappssmtp.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id pTTLhWBnnU_0 for ; Wed, 13 Jul 2016 22:25:58 +0000 (UTC) Received: from mail-pa0-f42.google.com (mail-pa0-f42.google.com [209.85.220.42]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 27DA65F242 for ; Wed, 13 Jul 2016 22:25:58 +0000 (UTC) Received: by mail-pa0-f42.google.com with SMTP id fi15so21576402pac.1 for ; Wed, 13 Jul 2016 15:25:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=occamsmachete-com.20150623.gappssmtp.com; s=20150623; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=Wk+cbaR+S5Ps7/uO0RvSKXI6JJnaMQbxvNX9qcVbFAs=; b=0NAsNFwJr6cgTgZiNj6/djx8uPrAm8C22sZ8wKAwDHjIURpKhLogs56V534Ii42TQX OJV5jykm+e2XUko6eQFXXKAiMXm1V8DBmnzUXpnMa1Gol2Xh/j9+EddZibBW/rNXcipB uJEQtdu8Epvne1WeeXRxzGcAU+2DM+q/5yksAanL3pBE9lI9oukRSFQOD3kHvIo9793O WVyzifYiqZOtX2tKjfDr6GOLCzHG4LjFcixGh776BlvZ2+0Kkz9YZagJNLHxmBli8OTI q2c+yTk6skNNOJdcFhdYGd5Ww6HDVAGgNgx3ULBdH17xwbltoDoPOaU4Kb3HRDNkbrGM HzZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=Wk+cbaR+S5Ps7/uO0RvSKXI6JJnaMQbxvNX9qcVbFAs=; b=ffhvH0oAbmsxrz+ji36bR2GEyQQxyL252GqhP0EWo74veFM6R6MS/00JmB2ERABy0V jo1p2AMCCJrvpIxYOv0GEhdnEl9rpWN479LG4YdUQSN82Os+UN8+Bxx1JhXtNzZeQ1im bmkBML1SCCjA4/j6DBJiOqXPJc0MiGxrA3GU5G7po3xHSBuLU3uhbS3pO5JzkAokNjAR TZNTn9+foqjeUwrYgHntOl81I7999jer5fnetP63fIJ/9c71dZhzuEE6W7dTHomMDB0k BfuN2ApRP4OKoXYkpqUSzA3Bwb53KQR8TrgRsCP5lPC4UbrbcCol/kdQmxQf+vwWu6qi iJOw== X-Gm-Message-State: ALyK8tIWm9KIqhNi+KemN6PEUlkJUF5BnrU7vHI75sxYx6RcP9PVro+CIHcczj1s8UJgKQ== X-Received: by 10.66.226.201 with SMTP id ru9mr17247718pac.55.1468448751317; Wed, 13 Jul 2016 15:25:51 -0700 (PDT) Received: from [192.168.0.2] (c-50-181-210-153.hsd1.wa.comcast.net. [50.181.210.153]) by smtp.gmail.com with ESMTPSA id j21sm1553958pfj.75.2016.07.13.15.25.49 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 13 Jul 2016 15:25:49 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_34A78847-8D4D-48DC-9B57-4B2E2D01EE15" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HBase warn on startup From: Pat Ferrel In-Reply-To: Date: Wed, 13 Jul 2016 15:25:48 -0700 Cc: user@predictionio.incubator.apache.org Message-Id: References: To: Tom Chan X-Mailer: Apple Mail (2.3124) archived-at: Wed, 13 Jul 2016 22:26:03 -0000 --Apple-Mail=_34A78847-8D4D-48DC-9B57-4B2E2D01EE15 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Something to do with swapping the location of data to a new AWS volume = seems to have caused it and the command below does seem to have fixed = it, will watch to see if it resurfaces. Thanks Tom! BTW pio works with HBase 1.2.1 On Jul 13, 2016, at 3:02 PM, Tom Chan wrote: I did not have time to dig into this when it happened, but HBase is = somehow corrupted where a=20 hbase hbck -fixEmptyMetaCells is a temporary fix. It happened for only one client but I'm not sure = what caused it. = http://mail-archives.apache.org/mod_mbox/hbase-user/201505.mbox/%3CCAEP_jW= bf1KGhmsL272HymTODSKtS68TP6b3Lh2ECSQ-NWE9_EQ@mail.gmail.com%3E = On Wed, Jul 13, 2016 at 2:42 PM, Pat Ferrel > wrote: I saw a question Tom asked about this error when doing `pio status` Was = there every an answer for what causes it? [INFO] [Storage$] Verifying Event Data Backend (Source: HBASE)... [INFO] [Storage$] Test writing to Event Store (App Id 0)... [WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [INFO] [HBLEvents] The table pio_event:events_0 doesn't exist yet. = Creating now... [WARN] [HBaseAdmin] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [WARN] [HBaseAdmin] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [WARN] [HBaseAdmin] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [WARN] [HBaseAdmin] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [INFO] [HBLEvents] Removing table pio_event:events_0... [WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0} [INFO] [Console$] (sleeping 5 seconds for all messages to show up...) [INFO] [Console$] Your system is all ready to go. --Apple-Mail=_34A78847-8D4D-48DC-9B57-4B2E2D01EE15 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Something to do with swapping the location of data to a new = AWS volume seems to have caused it and the command below does seem to = have fixed it, will watch to see if it resurfaces.

Thanks Tom!

BTW pio works with HBase = 1.2.1

On Jul = 13, 2016, at 3:02 PM, Tom Chan <tom.chan@salesforce.com> wrote:

I did not have time to dig into this when it = happened, but HBase is somehow corrupted where a 
 hbase hbck -fixEmptyMetaCells
is a temporary fix. It = happened for only one client but I'm not sure what caused = it.

http://mail-archives.apache.org/mod_mbox/hbase-user/201505.mbox= /%3CCAEP_jWbf1KGhmsL272HymTODSKtS68TP6b3Lh2ECSQ-NWE9_EQ@mail.gmail.com%3E<= /a>

On Wed, = Jul 13, 2016 at 2:42 PM, Pat Ferrel <pat@occamsmachete.com> wrote:
I saw a question Tom = asked about this error when doing `pio status` Was there every an answer = for what causes it?

[INFO] [Storage$] Verifying Event Data Backend (Source: HBASE)...
[INFO] [Storage$] Test writing to Event Store (App Id 0)...
= [WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[INFO] [HBLEvents] The table pio_event:events_0 doesn't exist yet. = Creating now...
[WARN] [HBaseAdmin] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[WARN] [HBaseAdmin] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[WARN] [HBaseAdmin] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[WARN] [HBaseAdmin] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[INFO] [HBLEvents] Removing table pio_event:events_0...
[WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[WARN] [MetaReader] No serialized HRegionInfo in = keyvalues=3D{pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea= 55377./info:seqnumDuringOpen/1468443836320/Put/vlen=3D8/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erver/1468443836320/Put/vlen=3D21/mvcc=3D0, = pio_event:events_0,,1468443835768.05bd150642e05c034a70d516fea55377./info:s= erverstartcode/1468443836320/Put/vlen=3D8/mvcc=3D0}
[INFO] [Console$] (sleeping 5 seconds for all messages to show up...)
[INFO] [Console$] Your system is all ready to go.



= --Apple-Mail=_34A78847-8D4D-48DC-9B57-4B2E2D01EE15--