From dev-return-67494-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Thu Jan 11 03:38:51 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id DBAF718072F for ; Thu, 11 Jan 2018 03:38:51 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id CBA8A160C40; Thu, 11 Jan 2018 02:38:51 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6FE1A160C2E for ; Thu, 11 Jan 2018 03:38:48 +0100 (CET) Received: (qmail 23713 invoked by uid 500); 11 Jan 2018 02:38:45 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 23690 invoked by uid 99); 11 Jan 2018 02:38:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Jan 2018 02:38:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C3F781808D1 for ; Thu, 11 Jan 2018 02:38:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.881 X-Spam-Level: * X-Spam-Status: No, score=1.881 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 50jFhaqMexia for ; Thu, 11 Jan 2018 02:38:21 +0000 (UTC) Received: from mail-ua0-f182.google.com (mail-ua0-f182.google.com [209.85.217.182]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E415F5F39E for ; Thu, 11 Jan 2018 02:38:20 +0000 (UTC) Received: by mail-ua0-f182.google.com with SMTP id g16so712101ual.2 for ; Wed, 10 Jan 2018 18:38:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=8Rb2KOPC0cT8gd5TX8s0Zd5y/aoOAuUXfE9vwqI/SVU=; b=fmq6rgCLt0a2Yj3fr/FX7f24700voef50H05KqFQ7SY7ic8+eJHPSTZEvxIRLyLLXu NGtCFRYnnjEqYCKhcYkCNx1TOsA6gFBNfc+Y8PvN9ZqyVGLsSHtQVirXjioITBCKsbJE v4Q4Qh/FKH0OfyoU2t/jYNb6sXWbhLadd2EVAOdi4BPSmwBGVnLw/SRyatuKIRQ7dLLZ VPJMqjQxexnT5RGT9d4Pejupu6yKjjmu8UmM8fEYKuevut2Z6d3BMKNEBB/12omyg4Dc ws6f2B+ebK2qWLN4IlgHnAPRuFPWqqdFxW4/B2mJIXJHbq+f08xH9cv0qBwheog1rwf9 F/Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=8Rb2KOPC0cT8gd5TX8s0Zd5y/aoOAuUXfE9vwqI/SVU=; b=OCJjdzrQXxVjHfFgcG7H60enaCOU3jahrsFmDv/La/lFWL/NzBz7l/4DHPZgjP8/JY T4ephPQqcVY2v8iz5L329Lw3nE9E7lGZ3HcI9l86XZ/5zvSeftUK6vFfVxTyBEk0E3qR CHB2naZlduXldfqC50aefxBB/3hoqgewkm3NSHg+VvifKR0QaS94A1DLUWxXLQnBlxtI AhB9TXmu3m6e/3iuCpbfLEw1ae7x362Qa0LhBQuh5tpjdPC+5T9Dd9TXiAmoM8StRdVM zVBM8LU393znKXB9sb0Kr5v7G/RVp81QohyDfYBYc+ZKnXzerJPCPw/7PFlmAVsHwxu8 +yiA== X-Gm-Message-State: AKwxytfp3ygDK5yyivdy4HTrrOsd6pp1BUu/mZJuhc4Ur3DJOhtA14EM cmG0lRJ4NageOPTQDN2CC3jLzI+PpwICbVDB58nx/Lo= X-Google-Smtp-Source: ACJfBouBz1i0gjU54BZQGMlXAsPNI7PpusztVIhL2KSLlF2Stbq+6Fl9XkuKWc4qELjrYLY/L1rdoDtptZQgGFCTIHE= X-Received: by 10.176.21.165 with SMTP id i34mr6749339uae.108.1515638299438; Wed, 10 Jan 2018 18:38:19 -0800 (PST) MIME-Version: 1.0 Received: by 10.31.49.142 with HTTP; Wed, 10 Jan 2018 18:37:48 -0800 (PST) In-Reply-To: References: From: Apekshit Sharma Date: Wed, 10 Jan 2018 18:37:48 -0800 Message-ID: Subject: Re: [VOTE] The first hbase-2.0.0-beta-1 Release Candidate is available To: dev@hbase.apache.org Content-Type: multipart/alternative; boundary="001a1144402a7254700562770a39" --001a1144402a7254700562770a39 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Jan 10, 2018 at 11:25 AM, Zach York wrote: > What is the expectation for flaky tests? I was going to post some test > failures, but saw that they were included in the excludes for flaky tests= . > > I understand we might be okay with having flaky tests for this beta-1 (an= d > obviously for dev), but I would assume that we want consistent test resul= ts > for the official 2.0.0 release. > Yeah, that's the goal, but sadly not many hands on deck are working on that, so doesn't seem in reach. > Do we have JIRAs created for all the flaky tests so that we can start > fixing them before the beta-2/official RCs get put up? > Whenever i start working on one, i search for it in jira first in case someone's already working on it, if not I create a new one. (treating jira as a lock to avoid redundant work). Creating just the jiras doesn't really help until someone takes them and so most just remain open. But chicken and egg problem maybe? Might be good to create jira for a few simple ones to see if peeps starting contributing on this front? > I'd be happy to help try to track down the root causes of the flakiness a= nd > try to fix these problematic tests. > Any help here would be great! Here's a personal thank you : http://calmcoolcollective.net/wp-content/uploads/2016/08/chocolatechip.jpg :) > > Thanks, > Zach > > On Wed, Jan 10, 2018 at 9:37 AM, Stack wrote: > > > Put up a JIRA and dump this stuff in JMS. Sounds like we need a bit mor= e > > test coverage at least. Thanks sir. > > St.Ack > > > > On Wed, Jan 10, 2018 at 2:52 AM, Jean-Marc Spaggiari < > > jean-marc@spaggiari.org> wrote: > > > > > The DN was dead since December 31st... I really hope the DN figured > that > > > :-/ > > > > > > I will retry with making sure that the NN is aware the local DN is > dead, > > > and see. I let you know. > > > > > > Thanks, > > > > > > JMS > > > > > > 2018-01-10 5:50 GMT-05:00 =E5=BC=A0=E9=93=8E(Duo Zhang) : > > > > > > > The problem maybe that the DN is dead, but NN does not know and kee= p > > > > telling RS that you should try to connect to it. And for the new > > > > AsyncFSWAL, we need to connect to all the 3 DNs successfully befor= e > > > > writing actual data to it, so the RS sucks... > > > > > > > > This maybe a problem. > > > > > > > > 2018-01-10 18:40 GMT+08:00 Jean-Marc Spaggiari < > > jean-marc@spaggiari.org > > > >: > > > > > > > > > You're correct. It was dead. I thought HBase will be able to > survive > > > > that. > > > > > Same the DN dies after the RS has started, RS will fail closing > > nicely > > > :( > > > > > > > > > > 2018-01-10 5:38 GMT-05:00 =E5=BC=A0=E9=93=8E(Duo Zhang) : > > > > > > > > > > > Connection refuse? Have you checked the status of the datanode = on > > > > node8? > > > > > > > > > > > > 2018-01-10 18:31 GMT+08:00 Jean-Marc Spaggiari < > > > > jean-marc@spaggiari.org > > > > > >: > > > > > > > > > > > > > I know, this one sunk, but still running it on my cluster, so > > here > > > > is a > > > > > > new > > > > > > > issue I just got.... > > > > > > > > > > > > > > Any idea what this can be? I see this only a one of my nodes.= .. > > > > > > > > > > > > > > 2018-01-10 05:22:55,786 WARN [regionserver/node8.com/192. > > > > > 168.23.2:16020 > > > > > > ] > > > > > > > wal.AsyncFSWAL: create wal log writer hdfs:// > > > > > > > node2.com:8020/hbase/WALs/node8.com,16020,1515579724994/ > > > > > > node8.com%2C16020% > > > > > > > 2C1515579724994.1515579743134 > > > > > > > failed, retry =3D 6 > > > > > > > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$ > > > > > > > AnnotatedConnectException: > > > > > > > syscall:getsockopt(..) failed: Connexion refus=C3=A9e: / > > > > 192.168.23.2:50010 > > > > > > > at > > > > > > > org.apache.hbase.thirdparty.io.netty.channel.unix.Socket. > > > > > > > finishConnect(..)(Unknown > > > > > > > Source) > > > > > > > Caused by: > > > > > > > org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$ > > > > > > > NativeConnectException: > > > > > > > syscall:getsockopt(..) failed: Connexion refus=C3=A9e > > > > > > > ... 1 more > > > > > > > > > > > > > > > > > > > > > From the same node, if I ls while the RS is starting, I can s= ee > > the > > > > > > related > > > > > > > directoy: > > > > > > > > > > > > > > > > > > > > > hbase@node8:~/hbase-2.0.0-beta-1/logs$ > > > > /home/hadoop/hadoop-2.7.5/bin/ > > > > > > hdfs > > > > > > > dfs -ls /hbase/WALs/ > > > > > > > Found 35 items > > > > > > > ... > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node1.com,16020,1515579724884 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node3.com,16020,1515579738916 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node4.com,16020,1515579717193 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node5.com,16020,1515579724586 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node6.com,16020,1515579724999 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node7.com,16020,1515579725681 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:23 > > > > > /hbase/WALs/ > > > > > > > node8.com,16020,1515579724994 > > > > > > > > > > > > > > > > > > > > > > > > > > > > and after the RS tries many times and fails the directory is > > gone: > > > > > > > hbase@node8:~/hbase-2.0.0-beta-1/logs$ > > > > /home/hadoop/hadoop-2.7.5/bin/ > > > > > > hdfs > > > > > > > dfs -ls /hbase/WALs/ > > > > > > > Found 34 items > > > > > > > ... > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node1.com,16020,1515579724884 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node3.com,16020,1515579738916 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node4.com,16020,1515579717193 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node5.com,16020,1515579724586 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node6.com,16020,1515579724999 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:22 > > > > > /hbase/WALs/ > > > > > > > node7.com,16020,1515579725681 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2018-01-10 05:23:46,177 ERROR [regionserver/node8.com/192. > > > > > 168.23.2:16020 > > > > > > ] > > > > > > > regionserver.HRegionServer: ***** ABORTING region server > > > > > > > node8.com,16020,1515579724994: > > > > > > > Unhandled: Failed to create wal log writer hdfs:// > > > > > > > node2.com:8020/hbase/WALs/node8.com,16020,1515579724994/ > > > > > > node8.com%2C16020% > > > > > > > 2C1515579724994.1515579743134 > > > > > > > after retrying 10 time(s) ***** > > > > > > > java.io.IOException: Failed to create wal log writer hdfs:// > > > > > > > node2.com:8020/hbase/WALs/node8.com,16020,1515579724994/ > > > > > > node8.com%2C16020% > > > > > > > 2C1515579724994.1515579743134 > > > > > > > after retrying 10 time(s) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL. > > > > > > createWriterInstance( > > > > > > > AsyncFSWAL.java:663) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL. > > > > > > createWriterInstance( > > > > > > > AsyncFSWAL.java:130) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.wal. > > AbstractFSWAL.rollWriter( > > > > > > > AbstractFSWAL.java:766) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.wal. > > AbstractFSWAL.rollWriter( > > > > > > > AbstractFSWAL.java:504) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.< > > > > > > > init>(AsyncFSWAL.java:264) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL( > > > > > > > AsyncFSWALProvider.java:69) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL( > > > > > > > AsyncFSWALProvider.java:44) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL( > > > > > > > AbstractFSWALProvider.java:139) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL( > > > > > > > AbstractFSWALProvider.java:55) > > > > > > > at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory. > > > > java:244) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer. > > > > > > > getWAL(HRegionServer.java:2123) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer. > > > > > > > buildServerLoad(HRegionServer.java:1315) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer. > > > > > > tryRegionServerReport( > > > > > > > HRegionServer.java:1196) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer. > > > > > > > run(HRegionServer.java:1008) > > > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > > > > > > > 2018-01-10 05:23:46,324 INFO [regionserver/node8.com/192. > > > > > 168.23.2:16020 > > > > > > ] > > > > > > > regionserver.HRegionServer: regionserver/node8.com/192. > > > > 168.23.2:16020 > > > > > > > exiting > > > > > > > 2018-01-10 05:23:46,324 ERROR [main] regionserver. > > > > > > > HRegionServerCommandLine: > > > > > > > Region server exiting > > > > > > > java.lang.RuntimeException: HRegionServer Aborted > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine= . > > > start( > > > > > > > HRegionServerCommandLine.java:66) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver. > > HRegionServerCommandLine.run( > > > > > > > HRegionServerCommandLine.java:85) > > > > > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain( > > > > > > > ServerCommandLine.java:149) > > > > > > > at > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer. > > > > > > > main(HRegionServer.java:3016) > > > > > > > > > > > > > > Which is very surprising, because I can clearly see the > directory > > > > being > > > > > > > created. > > > > > > > > > > > > > > Another attempt here, we I even look one step deeper and can > see > > > the > > > > > > > generated file: > > > > > > > 2018-01-10 05:27:58,116 WARN [regionserver/node8.com/192. > > > > > 168.23.2:16020 > > > > > > ] > > > > > > > wal.AsyncFSWAL: create wal log writer > > > > > > > hdfs://node2.com:8020*/hbase/WALs/node8.com > > > > > > > ,16020,1515580031417/node8.com > > > > > > > %2C16020%2C1515580031417.1515580037373* > > failed, > > > > > retry > > > > > > =3D > > > > > > > 7 > > > > > > > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$ > > > > > > > AnnotatedConnectException: > > > > > > > syscall:getsockopt(..) failed: Connexion refus=C3=A9e: / > > > > 192.168.23.2:50010 > > > > > > > at > > > > > > > org.apache.hbase.thirdparty.io.netty.channel.unix.Socket. > > > > > > > finishConnect(..)(Unknown > > > > > > > Source) > > > > > > > Caused by: > > > > > > > org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$ > > > > > > > NativeConnectException: > > > > > > > syscall:getsockopt(..) failed: Connexion refus=C3=A9e > > > > > > > ... 1 more > > > > > > > 2018-01-10 05:28:08,210 INFO [regionserver/node8.com/192. > > > > > 168.23.2:16020 > > > > > > ] > > > > > > > util.FSHDFSUtils: Recover lease on dfs file /hbase/WALs/ > > node8.com > > > > > > > ,16020,1515580031417/node8.com%2C16020%2C1515580031417. > > > 1515580037373 > > > > > > > 2018-01-10 05:28:08,228 INFO [regionserver/node8.com/192. > > > > > 168.23.2:16020 > > > > > > ] > > > > > > > util.FSHDFSUtils: Failed to recover lease, attempt=3D0 on > > > > > file=3D/hbase/WALs/ > > > > > > > node8.com,16020,1515580031417/node8.com%2C16020% > > > > > > > 2C1515580031417.1515580037373 > > > > > > > after 17ms > > > > > > > > > > > > > > hbase@node8:~/hbase-2.0.0-beta-1/logs$ > > > > /home/hadoop/hadoop-2.7.5/bin/ > > > > > > hdfs > > > > > > > dfs -ls -R /hbase/WALs/ | grep node8 > > > > > > > drwxr-xr-x - hbase supergroup 0 2018-01-10 05:28 > > > > > /hbase/WALs/ > > > > > > > node8.com,16020,1515580031417 > > > > > > > -rw-r--r-- 3 hbase supergroup 0 2018-01-10 05:28 > > > > > > > */hbase/WALs/node8.com > > > > > > > ,16020,1515580031417/node8.com > > > > > > > %2C16020%2C1515580031417.1515580037373* > > > > > > > > > > > > > > > > > > > > > But still says it fails. Any clue? all other nodes are workin= g > > > fine. > > > > > > > > > > > > > > 2018-01-09 16:25 GMT-05:00 Stack : > > > > > > > > > > > > > > > On Tue, Jan 9, 2018 at 10:07 AM, Andrew Purtell < > > > > apurtell@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I just vetoed the RC because TestMemstoreLABWithoutPool > > always > > > > > fails > > > > > > > for > > > > > > > > > me. It was the same with the last RC too. My Java is Orac= le > > > Java > > > > > > 8u144 > > > > > > > > > running on x64 Linux (Ubuntu xenial). Let me know if you > need > > > me > > > > to > > > > > > > > provide > > > > > > > > > the test output. > > > > > > > > > > > > > > > > > > > > > > > > > > Ok. I can't make it fail. I'm going to disable it and file = an > > > issue > > > > > > where > > > > > > > > we can work on figuring what is different here. > > > > > > > > > > > > > > > > Thanks A, > > > > > > > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 9, 2018 at 9:31 AM, Stack > > > wrote: > > > > > > > > > > > > > > > > > > > I put up a new RC JMS. It still has flakies (though Duo > > fixed > > > > > > > > > > TestFromClientSide...). Was thinking that we could > release > > > > beta-1 > > > > > > > > though > > > > > > > > > it > > > > > > > > > > has flakies. We'll keep working on cutting these down a= s > we > > > > > > approach > > > > > > > > GA. > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > On Sun, Jan 7, 2018 at 10:02 PM, Stack > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > On Sun, Jan 7, 2018 at 3:14 AM, Jean-Marc Spaggiari < > > > > > > > > > > > jean-marc@spaggiari.org> wrote: > > > > > > > > > > > > > > > > > > > > > >> Ok, thanks Stack. I will keep it running all day lon= g > > > until > > > > I > > > > > > get > > > > > > > a > > > > > > > > > > >> successful one. Is that useful that I report all the > > > failed? > > > > > Or > > > > > > > > just a > > > > > > > > > > >> wast > > > > > > > > > > >> of time? Here is the last failed: > > > > > > > > > > >> > > > > > > > > > > >> [INFO] Results: > > > > > > > > > > >> [INFO] > > > > > > > > > > >> [ERROR] Failures: > > > > > > > > > > >> [ERROR] TestFromClientSide. > > > testCheckAndDeleteWithCompareO > > > > > > p:4982 > > > > > > > > > > >> expected: but was: > > > > > > > > > > >> [ERROR] Errors: > > > > > > > > > > >> [ERROR] TestDLSAsyncFSWAL>AbstractTestDLS. > > > > > > testThreeRSAbort:401 > > > > > > > =C2=BB > > > > > > > > > > >> TableNotFound Region ... > > > > > > > > > > >> [INFO] > > > > > > > > > > >> [ERROR] Tests run: 3585, Failures: 1, Errors: 1, > > Skipped: > > > 44 > > > > > > > > > > >> [INFO] > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > Thanks for bringing up flakies. If we look at the > > > nightlies' > > > > > run, > > > > > > > we > > > > > > > > > can > > > > > > > > > > > get the current list. Probably no harm if all tests > pass > > > once > > > > > in > > > > > > a > > > > > > > > > while > > > > > > > > > > > (smile). > > > > > > > > > > > > > > > > > > > > > > Looking at your findings, TestFromClientSide. > > > > > > > > > > testCheckAndDeleteWithCompareOp > > > > > > > > > > > looks to be new to beta-1. Its a cranky one. I'm > looking > > at > > > > it. > > > > > > > Might > > > > > > > > > > punt > > > > > > > > > > > to beta-2 if can't figure it by tomorrow. HBASE-19731= . > > > > > > > > > > > > > > > > > > > > > > TestDLSAsyncFSWAL is a flakey that unfortunately pass= es > > > > > locally. > > > > > > > > > > > > > > > > > > > > > > Let me see what others we have... > > > > > > > > > > > > > > > > > > > > > > S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> JMS > > > > > > > > > > >> > > > > > > > > > > >> 2018-01-07 1:55 GMT-05:00 Apekshit Sharma < > > > > appy@cloudera.com > > > > > >: > > > > > > > > > > >> > > > > > > > > > > >> > bq. Don't you think we have enough branches alread= y > > > mighty > > > > > > Appy? > > > > > > > > > > >> > Yeah we do...sigh. > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > idk about that. But don't we need a *patch* branch > > > > > branch-2.0 > > > > > > > > (just > > > > > > > > > > like > > > > > > > > > > >> > branch-1.4) where we "make backwards-compatible bu= g > > > fixes" > > > > > > and a > > > > > > > > > > *minor* > > > > > > > > > > >> > branch branch-2 where we "add functionality in a > > > > > > > > > backwards-compatible > > > > > > > > > > >> > manner". > > > > > > > > > > >> > Quotes are from http://hbase.apache.org/book.h > > > > > > > > > > >> tml#hbase.versioning.post10. > > > > > > > > > > >> > I stumbled on this issue when thinking about > > backporting > > > > > > > > > > >> > https://issues.apache.org/jira/browse/HBASE-17436 > for > > > > 2.1. > > > > > > > > > > >> > > > > > > > > > > > >> > -- Appy > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > On Sat, Jan 6, 2018 at 4:11 PM, stack < > > > > saint.ack@gmail.com> > > > > > > > > wrote: > > > > > > > > > > >> > > > > > > > > > > > >> > > It is not you. There are a bunch of flies we ne= ed > > to > > > > fix. > > > > > > > This > > > > > > > > > > >> latter is > > > > > > > > > > >> > > for sure flakey. Let me take a look. Thanks, JM= S. > > > > > > > > > > >> > > > > > > > > > > > > >> > > S > > > > > > > > > > >> > > > > > > > > > > > > >> > > On Jan 6, 2018 5:57 PM, "Jean-Marc Spaggiari" < > > > > > > > > > > >> jean-marc@spaggiari.org> > > > > > > > > > > >> > > wrote: > > > > > > > > > > >> > > > > > > > > > > > > >> > > I might not doing the right magic to get that > > run.... > > > If > > > > > > > someone > > > > > > > > > is > > > > > > > > > > >> able > > > > > > > > > > >> > to > > > > > > > > > > >> > > get all the tests pass, can you please share the > > > command > > > > > you > > > > > > > > run? > > > > > > > > > > >> > > > > > > > > > > > > >> > > Thanks, > > > > > > > > > > >> > > > > > > > > > > > > >> > > JMS > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > [INFO] Results: > > > > > > > > > > >> > > [INFO] > > > > > > > > > > >> > > [ERROR] Failures: > > > > > > > > > > >> > > [ERROR] TestFromClientSide. > > > > > testCheckAndDeleteWithCompareO > > > > > > > > p:4982 > > > > > > > > > > >> > > expected: but was: > > > > > > > > > > >> > > [ERROR] > > > > > > > > > > >> > > org.apache.hadoop.hbase.master.assignment. > > > > > > TestMergeTableRegi > > > > > > > > > > >> onsProcedure > > > > > > > > > > >> > > .testMergeRegionsConcurrently( > > > org.apache.hadoop.hbase. > > > > > > > > > master.assig > > > > > > > > > > >> > > nment.TestMergeTableRegionsProcedure) > > > > > > > > > > >> > > [ERROR] Run 1: > > > > > > > > > > >> > > TestMergeTableRegionsProcedure > > > > > > .setup:111->resetProcExecutorT > > > > > > > > > > >> estingKillFl > > > > > > > > > > >> > > ag:138 > > > > > > > > > > >> > > expected executor to be running > > > > > > > > > > >> > > [ERROR] Run 2: > > > > > > > > > > >> > > TestMergeTableRegionsProcedure.tearDown:128-> > > > > > > > > > > >> > > resetProcExecutorTestingKillFl > > > > > > > > > > >> > > ag:138 > > > > > > > > > > >> > > expected executor to be running > > > > > > > > > > >> > > [INFO] > > > > > > > > > > >> > > [ERROR] > > > > > > > > > > >> > > org.apache.hadoop.hbase.master.assignment. > > > > > > TestMergeTableRegi > > > > > > > > > > >> onsProcedure > > > > > > > > > > >> > > .testMergeTwoRegions(org. > > apache.hadoop.hbase.master. > > > > > > > > > assignment.Tes > > > > > > > > > > >> > > tMergeTableRegionsProcedure) > > > > > > > > > > >> > > [ERROR] Run 1: > > > > > > > > > > >> > > TestMergeTableRegionsProcedure > > > > > > .setup:111->resetProcExecutorT > > > > > > > > > > >> estingKillFl > > > > > > > > > > >> > > ag:138 > > > > > > > > > > >> > > expected executor to be running > > > > > > > > > > >> > > [ERROR] Run 2: > > > > > > > > > > >> > > TestMergeTableRegionsProcedure.tearDown:128-> > > > > > > > > > > >> > > resetProcExecutorTestingKillFl > > > > > > > > > > >> > > ag:138 > > > > > > > > > > >> > > expected executor to be running > > > > > > > > > > >> > > [INFO] > > > > > > > > > > >> > > [ERROR] > > > > > > > > > > >> > > org.apache.hadoop.hbase.master.assignment. > > > > > > TestMergeTableRegi > > > > > > > > > > >> onsProcedure > > > > > > > > > > >> > . > > > > > > > > > > >> > > testRecoveryAndDoubleExecution > > > (org.apache.hadoop.hbase. > > > > > > > > master.ass > > > > > > > > > > >> > > ignment.TestMergeTableRegionsProcedure) > > > > > > > > > > >> > > [ERROR] Run 1: > > > > > > > > > > >> > > TestMergeTableRegionsProcedure > > > > > > .setup:111->resetProcExecutorT > > > > > > > > > > >> estingKillFl > > > > > > > > > > >> > > ag:138 > > > > > > > > > > >> > > expected executor to be running > > > > > > > > > > >> > > [ERROR] Run 2: > > > > > > > > > > >> > > TestMergeTableRegionsProcedure.tearDown:128-> > > > > > > > > > > >> > > resetProcExecutorTestingKillFl > > > > > > > > > > >> > > ag:138 > > > > > > > > > > >> > > expected executor to be running > > > > > > > > > > >> > > [INFO] > > > > > > > > > > >> > > [ERROR] > > > > > > > > > > >> > > org.apache.hadoop.hbase.master.assignment. > > > > > > TestMergeTableRegi > > > > > > > > > > >> onsProcedure > > > > > > > > > > >> > . > > > > > > > > > > >> > > testRollbackAndDoubleExecution > > > (org.apache.hadoop.hbase. > > > > > > > > master.ass > > > > > > > > > > >> > > ignment.TestMergeTableRegionsProcedure) > > > > > > > > > > >> > > [ERROR] Run 1: > > > > > > > > > > >> > > TestMergeTableRegionsProcedure. > > > > > > testRollbackAndDoubleExecution > > > > > > > > :272 > > > > > > > > > > >> > > expected: but was: > > > > > > > > > > >> > > [ERROR] Run 2: > > > > > > > > > > >> > > TestMergeTableRegionsProcedure.tearDown:128-> > > > > > > > > > > >> > > resetProcExecutorTestingKillFl > > > > > > > > > > >> > > ag:138 > > > > > > > > > > >> > > expected executor to be running > > > > > > > > > > >> > > [INFO] > > > > > > > > > > >> > > [ERROR] TestSnapshotQuotaObserverChore > > > > > > .testSnapshotSize:276 > > > > > > > > > > Waiting > > > > > > > > > > >> > > timed > > > > > > > > > > >> > > out after [30 000] msec > > > > > > > > > > >> > > [ERROR] > > > > > > > > > > >> > > TestHRegionWithInMemoryFlush>TestHRegion. > > > > > > testWritesWhileScan > > > > > > > > > > >> ning:3813 > > > > > > > > > > >> > > expected null, but was: > > > > > > > > > >> > NotServingRegionException: > > > > > > > > > > >> > > testWritesWhileScanning,,1515277468063. > > > > > > 468265483817cb6da6320 > > > > > > > > > > >> 26ba5b306f6. > > > > > > > > > > >> > > is > > > > > > > > > > >> > > closing> > > > > > > > > > > >> > > [ERROR] Errors: > > > > > > > > > > >> > > [ERROR] TestDLSAsyncFSWAL>AbstractTestDLS. > > > > > > > > testThreeRSAbort:401 > > > > > > > > > =C2=BB > > > > > > > > > > >> > > TableNotFound testThr... > > > > > > > > > > >> > > [ERROR] > > > > > > > > > > >> > > org.apache.hadoop.hbase.master.balancer. > > > > > > > > > TestRegionsOnMasterOptions. > > > > > > > > > > >> > > testRegionsOnAllServers(org. > > > apache.hadoop.hbase.master. > > > > > > > > balancer. > > > > > > > > > > >> > > TestRegionsOnMasterOptions) > > > > > > > > > > >> > > [ERROR] Run 1: > > > > > > > > > > >> > > TestRegionsOnMasterOptions. > > > testRegionsOnAllServers:94-> > > > > > > > > > > >> > > checkBalance:207->Object.wait:-2 > > > > > > > > > > >> > > =C2=BB TestTimedOut > > > > > > > > > > >> > > [ERROR] Run 2: TestRegionsOnMasterOptions. > > > > > > > > > testRegionsOnAllServers > > > > > > > > > > =C2=BB > > > > > > > > > > >> > > Appears to be stuck in t... > > > > > > > > > > >> > > [INFO] > > > > > > > > > > >> > > [INFO] > > > > > > > > > > >> > > [ERROR] Tests run: 3604, Failures: 7, Errors: 2, > > > > Skipped: > > > > > 44 > > > > > > > > > > >> > > [INFO] > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > 2018-01-06 15:52 GMT-05:00 Jean-Marc Spaggiari < > > > > > > > > > > >> jean-marc@spaggiari.org > > > > > > > > > > >> > >: > > > > > > > > > > >> > > > > > > > > > > > > >> > > > Deleted the class to get all the tests running= . > > Was > > > > > > running > > > > > > > on > > > > > > > > > the > > > > > > > > > > >> RC1 > > > > > > > > > > >> > > > from the tar. > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > I know get those one failing. > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > [ERROR] Failures: > > > > > > > > > > >> > > > [ERROR] TestFavoredStochasticLoadBalan > > > > > > > > > > >> cer.test2FavoredNodesDead:352 > > > > > > > > > > >> > > > Balancer did not run > > > > > > > > > > >> > > > [ERROR] TestRegionMergeTransactionOnCl > > > > > > > > > > >> uster.testCleanMergeReference: > > > > > > > > > > >> > > 284 > > > > > > > > > > >> > > > hdfs://localhost:45311/user/ > > > > > jmspaggi/test-data/7c269e83- > > > > > > > > > > >> > > > 5982-449e-8cf8-6babaaaa4c7c/data/default/ > > > > > > > > > testCleanMergeReference/ > > > > > > > > > > >> > > > f1bdc6441b090dbacb391c74eaf0d1d0 > > > > > > > > > > >> > > > [ERROR] Errors: > > > > > > > > > > >> > > > [ERROR] TestDLSAsyncFSWAL>AbstractTestDLS. > > > > > > > > > testThreeRSAbort:401 > > > > > > > > > > =C2=BB > > > > > > > > > > >> > > > TableNotFound Region ... > > > > > > > > > > >> > > > [INFO] > > > > > > > > > > >> > > > [ERROR] Tests run: 3604, Failures: 2, Errors: = 1, > > > > > Skipped: > > > > > > 44 > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > I have not been able to get all the tests pass= ed > > > > locally > > > > > > > for a > > > > > > > > > > >> while :( > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > JM > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > 2018-01-06 15:05 GMT-05:00 Ted Yu < > > > > yuzhihong@gmail.com > > > > > >: > > > > > > > > > > >> > > > > > > > > > > > > > >> > > >> Looks like you didn't include HBASE-19666 whi= ch > > > would > > > > > be > > > > > > in > > > > > > > > the > > > > > > > > > > >> next > > > > > > > > > > >> > RC. > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> On Sat, Jan 6, 2018 at 10:52 AM, Jean-Marc > > > Spaggiari > > > > < > > > > > > > > > > >> > > >> jean-marc@spaggiari.org> wrote: > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > Trying with a different command line (mvn > test > > -P > > > > > > > > runAllTests > > > > > > > > > > >> > > >> > -Dsurefire.secondPartThreadCount=3D12 > > > > > > > > > > >> -Dtest.build.data.basedirector > > > > > > > > > > >> > > >> y=3D/ram4g > > > > > > > > > > >> > > >> > ) I get all those one failing. How are you > > able > > > to > > > > > get > > > > > > > > > > >> everything > > > > > > > > > > >> > > >> passed??? > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > [INFO] Results: > > > > > > > > > > >> > > >> > [INFO] > > > > > > > > > > >> > > >> > [ERROR] Failures: > > > > > > > > > > >> > > >> > [ERROR] TestDefaultCompactSelection. > > > > > > > > > testCompactionRatio:74-> > > > > > > > > > > >> > TestCom > > > > > > > > > > >> > > >> > pactionPolicy.compactEquals: > > > > > 182->TestCompactionPolicy. > > > > > > > > > > >> > > compactEquals:201 > > > > > > > > > > >> > > >> > expected:<[[4, 2, 1]]> but was:<[[]]> > > > > > > > > > > >> > > >> > [ERROR] TestDefaultCompactSelection. > > > > > > > > > > >> > testStuckStoreCompaction:145->T > > > > > > > > > > >> > > >> > estCompactionPolicy.compactEquals:182-> > > > > > > > > TestCompactionPolicy. > > > > > > > > > > >> > > >> compactEquals:201 > > > > > > > > > > >> > > >> > expected:<[[]30, 30, 30]> but was:<[[99, 30= , > > ]30, > > > > 30, > > > > > > > 30]> > > > > > > > > > > >> > > >> > [INFO] > > > > > > > > > > >> > > >> > [ERROR] Tests run: 1235, Failures: 2, Error= s: > > 0, > > > > > > > Skipped: 4 > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > Second run: > > > > > > > > > > >> > > >> > [INFO] Results: > > > > > > > > > > >> > > >> > [INFO] > > > > > > > > > > >> > > >> > [ERROR] Failures: > > > > > > > > > > >> > > >> > [ERROR] TestDefaultCompactSelection. > > > > > > > > > testCompactionRatio:74-> > > > > > > > > > > >> > > >> > TestCompactionPolicy.compactEquals:182-> > > > > > > > > TestCompactionPolicy > > > > > > > > > > >> > > >> .compactEquals:201 > > > > > > > > > > >> > > >> > expected:<[[4, 2, 1]]> but was:<[[]]> > > > > > > > > > > >> > > >> > [ERROR] TestDefaultCompactSelection. > > > > > > > > > > >> > testStuckStoreCompaction:145-> > > > > > > > > > > >> > > >> > TestCompactionPolicy.compactEquals:182-> > > > > > > > > TestCompactionPolicy > > > > > > > > > > >> > > >> .compactEquals:201 > > > > > > > > > > >> > > >> > expected:<[[]30, 30, 30]> but was:<[[99, 30= , > > ]30, > > > > 30, > > > > > > > 30]> > > > > > > > > > > >> > > >> > [INFO] > > > > > > > > > > >> > > >> > [ERROR] Tests run: 1235, Failures: 2, Error= s: > > 0, > > > > > > > Skipped: 4 > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > Then again: > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > [INFO] Results: > > > > > > > > > > >> > > >> > [INFO] > > > > > > > > > > >> > > >> > [ERROR] Failures: > > > > > > > > > > >> > > >> > [ERROR] TestDefaultCompactSelection. > > > > > > > > > testCompactionRatio:74-> > > > > > > > > > > >> > > >> > TestCompactionPolicy.compactEquals:182-> > > > > > > > > TestCompactionPolicy > > > > > > > > > > >> > > >> .compactEquals:201 > > > > > > > > > > >> > > >> > expected:<[[4, 2, 1]]> but was:<[[]]> > > > > > > > > > > >> > > >> > [ERROR] TestDefaultCompactSelection. > > > > > > > > > > >> > testStuckStoreCompaction:145-> > > > > > > > > > > >> > > >> > TestCompactionPolicy.compactEquals:182-> > > > > > > > > TestCompactionPolicy > > > > > > > > > > >> > > >> .compactEquals:201 > > > > > > > > > > >> > > >> > expected:<[[]30, 30, 30]> but was:<[[99, 30= , > > ]30, > > > > 30, > > > > > > > 30]> > > > > > > > > > > >> > > >> > [INFO] > > > > > > > > > > >> > > >> > [ERROR] Tests run: 1235, Failures: 2, Error= s: > > 0, > > > > > > > Skipped: 4 > > > > > > > > > > >> > > >> > [INFO] > > > > > > > > > > >> > > >> > [INFO] ------------------------------ > > > > > > > > > > >> ------------------------------ > > > > > > > > > > >> > > >> > ------------ > > > > > > > > > > >> > > >> > [INFO] Reactor Summary: > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > Sound like it's always the exact same resul= t. > > Do > > > I > > > > > > have a > > > > > > > > way > > > > > > > > > > to > > > > > > > > > > >> > > exclude > > > > > > > > > > >> > > >> > this TestCompactionPolicy test from the run= ? > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > Here are more details from the last failure= : > > > > > > > > > > >> > > >> > ------------------------------ > > > > > > > > ------------------------------ > > > > > > > > > > >> > > >> > ------------------- > > > > > > > > > > >> > > >> > Test set: org.apache.hadoop.hbase. > > regionserver. > > > > > > > > > > >> > > TestDefaultCompactSelec > > > > > > > > > > >> > > >> tion > > > > > > > > > > >> > > >> > ------------------------------ > > > > > > > > ------------------------------ > > > > > > > > > > >> > > >> > ------------------- > > > > > > > > > > >> > > >> > Tests run: 4, Failures: 2, Errors: 0, > Skipped: > > 0, > > > > > Time > > > > > > > > > elapsed: > > > > > > > > > > >> > 1.323 > > > > > > > > > > >> > > s > > > > > > > > > > >> > > >> > <<< FAILURE! - in org.apache.hadoop.hbase. > > > > > > regionserver. > > > > > > > > > > >> > > >> > TestDefaultCompactSelection > > > > > > > > > > >> > > >> > testStuckStoreCompaction(org. > > > apache.hadoop.hbase. > > > > > > > > regionserve > > > > > > > > > > >> > > >> r.TestDefaultCompactSelection) > > > > > > > > > > >> > > >> > Time elapsed: 1.047 s <<< FAILURE! > > > > > > > > > > >> > > >> > org.junit.ComparisonFailure: expected:<[[]3= 0, > > 30, > > > > > 30]> > > > > > > > but > > > > > > > > > > >> > was:<[[99, > > > > > > > > > > >> > > >> 30, > > > > > > > > > > >> > > >> > ]30, 30, 30]> > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase. > > regionserver. > > > > > > > > > > >> > > >> > TestDefaultCompactSelection. > > > > > testStuckStoreCompaction( > > > > > > > > > > >> > > >> > TestDefaultCompactSelection.java:145) > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > testCompactionRatio(org.apache.hadoop.hbase= . > > > > > > > > regionserver.Tes > > > > > > > > > > >> > > >> tDefaultCompactSelection) > > > > > > > > > > >> > > >> > Time elapsed: 0.096 s <<< FAILURE! > > > > > > > > > > >> > > >> > org.junit.ComparisonFailure: expected:<[[4, > 2, > > > 1]]> > > > > > but > > > > > > > > > > >> was:<[[]]> > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase. > > regionserver. > > > > > > > > > > >> > > >> > TestDefaultCompactSelection. > > testCompactionRatio( > > > > > > > > > > >> > > >> > TestDefaultCompactSelection.java:74) > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > 2018-01-06 12:53:53,240 WARN > > > > > > > > [StoreOpener-22ce1d683ba4b6b93 > > > > > > > > > > >> > > >> 73a3c541ebab2a2-1] > > > > > > > > > > >> > > >> > util.CommonFSUtils(536): FileSystem doesn't > > > support > > > > > > > > > > >> > setStoragePolicy; > > > > > > > > > > >> > > >> > HDFS-6584, HDFS-9345 not available. This is > > > normal > > > > > and > > > > > > > > > expected > > > > > > > > > > >> on > > > > > > > > > > >> > > >> earlier > > > > > > > > > > >> > > >> > Hadoop versions. > > > > > > > > > > >> > > >> > java.lang.NoSuchMethodException: > > > > > org.apache.hadoop.fs. > > > > > > > > > > >> > > LocalFileSystem. > > > > > > > > > > >> > > >> > setStoragePolicy(org.apache.hadoop.fs.Path, > > > > > > > > > java.lang.String) > > > > > > > > > > >> > > >> > at java.lang.Class. > > > > > > getDeclaredMethod(Class.java: > > > > > > > > > 2130) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase.util. > > > > > CommonFSUtils. > > > > > > > > > > >> > > >> > invokeSetStoragePolicy( > CommonFSUtils.java:528) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase.util. > > > > > CommonFSUtils. > > > > > > > > > > >> > > setStoragePolicy( > > > > > > > > > > >> > > >> > CommonFSUtils.java:518) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase.region > > > > > > > > > > >> server.HRegionFileSystem. > > > > > > > > > > >> > > >> > setStoragePolicy(HRegionFileSystem.java:193= ) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase. > > > > > > > > > regionserver.HStore.( > > > > > > > > > > >> > > >> > HStore.java:250) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase. > > > > > > regionserver.HRegion. > > > > > > > > > > >> > > >> > instantiateHStore(HRegion.java:5497) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase. > > > > > > > > > > regionserver.HRegion$1.call( > > > > > > > > > > >> > > >> > HRegion.java:1002) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase. > > > > > > > > > > regionserver.HRegion$1.call( > > > > > > > > > > >> > > >> > HRegion.java:999) > > > > > > > > > > >> > > >> > at java.util.concurrent.FutureTas > > > > > > > > > > >> k.run(FutureTask.java:266) > > > > > > > > > > >> > > >> > at java.util.concurrent. > > > > > > > Executors$RunnableAdapter. > > > > > > > > > > >> > > >> > call(Executors.java:511) > > > > > > > > > > >> > > >> > at java.util.concurrent.FutureTas > > > > > > > > > > >> k.run(FutureTask.java:266) > > > > > > > > > > >> > > >> > at java.util.concurrent. > > > > > > > > > ThreadPoolExecutor.runWorker( > > > > > > > > > > >> > > >> > ThreadPoolExecutor.java:1149) > > > > > > > > > > >> > > >> > at java.util.concurrent. > > > > > > > > > ThreadPoolExecutor$Worker.run( > > > > > > > > > > >> > > >> > ThreadPoolExecutor.java:624) > > > > > > > > > > >> > > >> > at java.lang.Thread.run(Thread. > > java:748) > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > 2018-01-06 12:53:53,322 DEBUG [main] > > > > > > > > util.CommonFSUtils(538): > > > > > > > > > > >> > > FileSystem > > > > > > > > > > >> > > >> > doesn't support setStoragePolicy; HDFS-6584= , > > > > > HDFS-9345 > > > > > > > not > > > > > > > > > > >> > available. > > > > > > > > > > >> > > >> This > > > > > > > > > > >> > > >> > is normal and expected on earlier Hadoop > > > versions. > > > > > > > > > > >> > > >> > java.lang.NoSuchMethodException: > > > > > org.apache.hadoop.fs. > > > > > > > > > > >> > > LocalFileSystem. > > > > > > > > > > >> > > >> > setStoragePolicy(org.apache.hadoop.fs.Path, > > > > > > > > > java.lang.String) > > > > > > > > > > >> > > >> > at java.lang.Class. > > > > > > getDeclaredMethod(Class.java: > > > > > > > > > 2130) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase.util. > > > > > CommonFSUtils. > > > > > > > > > > >> > > >> > invokeSetStoragePolicy( > CommonFSUtils.java:528) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase.util. > > > > > CommonFSUtils. > > > > > > > > > > >> > > setStoragePolicy( > > > > > > > > > > >> > > >> > CommonFSUtils.java:518) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase.region > > > > > > > > > > >> server.HRegionFileSystem. > > > > > > > > > > >> > > >> > setStoragePolicy(HRegionFileSystem.java:193= ) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase. > > > > > > > > > regionserver.HStore.( > > > > > > > > > > >> > > >> > HStore.java:250) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase. > > regionserver. > > > > > > > > > > >> > TestCompactionPolicy. > > > > > > > > > > >> > > >> > initialize(TestCompactionPolicy.java:109) > > > > > > > > > > >> > > >> > at org.apache.hadoop.hbase. > > regionserver. > > > > > > > > > > >> > > >> > TestCompactionPolicy.setUp( > > > > > > TestCompactionPolicy.java:69) > > > > > > > > > > >> > > >> > at sun.reflect. > > NativeMethodAccessorImpl. > > > > > > > > > invoke0(Native > > > > > > > > > > >> > > Method) > > > > > > > > > > >> > > >> > at sun.reflect. > > NativeMethodAccessorImpl. > > > > > > invoke( > > > > > > > > > > >> > > >> > NativeMethodAccessorImpl.java:62) > > > > > > > > > > >> > > >> > at sun.reflect. > > > > DelegatingMethodAccessorImpl. > > > > > > > > invoke( > > > > > > > > > > >> > > >> > DelegatingMethodAccessorImpl.java:43) > > > > > > > > > > >> > > >> > at java.lang.reflect.Method. > > > > > > > > invoke(Method.java:498) > > > > > > > > > > >> > > >> > at org.junit.runners.model. > > > > > FrameworkMethod$1. > > > > > > > > > > >> > > runReflectiveCall( > > > > > > > > > > >> > > >> > FrameworkMethod.java:50) > > > > > > > > > > >> > > >> > at org.junit.internal.runners.mod > > > > > > > > > > >> el.ReflectiveCallable.run( > > > > > > > > > > >> > > >> > ReflectiveCallable.java:12) > > > > > > > > > > >> > > >> > at org.junit.runners.model. > > > > FrameworkMethod. > > > > > > > > > > >> > invokeExplosively( > > > > > > > > > > >> > > >> > FrameworkMethod.java:47) > > > > > > > > > > >> > > >> > at org.junit.internal.runners. > > > > > > > > statements.RunBefores. > > > > > > > > > > >> > > >> > evaluate(RunBefores.java:24) > > > > > > > > > > >> > > >> > at org.junit.internal.runners. > > > > > > > > > > >> > statements.RunAfters.evaluate( > > > > > > > > > > >> > > >> > RunAfters.java:27) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > ParentRunner.runLeaf( > > > > > > > > > > >> > > ParentRunner.java:325 > > > > > > > > > > >> > > >> ) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > BlockJUnit4ClassRunner. > > > > > > > > > runChild( > > > > > > > > > > >> > > >> > BlockJUnit4ClassRunner.java:78) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > BlockJUnit4ClassRunner. > > > > > > > > > runChild( > > > > > > > > > > >> > > >> > BlockJUnit4ClassRunner.java:57) > > > > > > > > > > >> > > >> > at org.junit.runners. > > ParentRunner$3.run( > > > > > > > > > > >> > > ParentRunner.java:290) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > ParentRunner$1.schedule( > > > > > > > > > > >> > > ParentRunner.java: > > > > > > > > > > >> > > >> 71) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > ParentRunner.runChildren( > > > > > > > > > > >> > > >> > ParentRunner.java:288) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > ParentRunner.access$000( > > > > > > > > > > >> > > ParentRunner.java: > > > > > > > > > > >> > > >> 58) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > ParentRunner$2.evaluate( > > > > > > > > > > >> > > >> > ParentRunner.java:268) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > > > > ParentRunner.run(ParentRunner. > > > > > > > > > > >> > java:363) > > > > > > > > > > >> > > >> > at org.junit.runners.Suite. > > > > > > > > runChild(Suite.java:128) > > > > > > > > > > >> > > >> > at org.junit.runners.Suite. > > > > > > > runChild(Suite.java:27) > > > > > > > > > > >> > > >> > at org.junit.runners. > > ParentRunner$3.run( > > > > > > > > > > >> > > ParentRunner.java:290) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > ParentRunner$1.schedule( > > > > > > > > > > >> > > ParentRunner.java: > > > > > > > > > > >> > > >> 71) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > ParentRunner.runChildren( > > > > > > > > > > >> > > >> > ParentRunner.java:288) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > ParentRunner.access$000( > > > > > > > > > > >> > > ParentRunner.java: > > > > > > > > > > >> > > >> 58) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > ParentRunner$2.evaluate( > > > > > > > > > > >> > > >> > ParentRunner.java:268) > > > > > > > > > > >> > > >> > at org.junit.runners. > > > > > > > > ParentRunner.run(ParentRunner. > > > > > > > > > > >> > java:363) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire. > > > > > > > > > junitcore.JUnitCore.run( > > > > > > > > > > >> > > >> > JUnitCore.java:55) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire. > > > > > > > > > > junitcore.JUnitCoreWrapper. > > > > > > > > > > >> > > >> > createRequestAndRun( > JUnitCoreWrapper.java:137) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire. > > > > > > > > > > junitcore.JUnitCoreWrapper. > > > > > > > > > > >> > > >> > executeEager(JUnitCoreWrapper.java:107) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire. > > > > > > > > > > junitcore.JUnitCoreWrapper. > > > > > > > > > > >> > > >> > execute(JUnitCoreWrapper.java:83) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire. > > > > > > > > > > junitcore.JUnitCoreWrapper. > > > > > > > > > > >> > > >> > execute(JUnitCoreWrapper.java:75) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire.juni > > > > > > > > > > >> tcore.JUnitCoreProvider. > > > > > > > > > > >> > > >> > invoke(JUnitCoreProvider.java:159) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire. > > > > > > > booter.ForkedBooter. > > > > > > > > > > >> > > >> > invokeProviderInSameClassLoade > > > > > r(ForkedBooter.java:373) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire. > > > > > > > booter.ForkedBooter. > > > > > > > > > > >> > > >> > runSuitesInProcess(ForkedBooter.java:334) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire.boot > > > > > > > > > > >> er.ForkedBooter.execute( > > > > > > > > > > >> > > >> > ForkedBooter.java:119) > > > > > > > > > > >> > > >> > at org.apache.maven.surefire. > > > > > > > > > booter.ForkedBooter.main( > > > > > > > > > > >> > > >> > ForkedBooter.java:407) > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > 2018-01-06 12:53:53,398 INFO [main] > > > > > > > > > > hbase.ResourceChecker(172): > > > > > > > > > > >> > > after: > > > > > > > > > > >> > > >> > regionserver.TestDefaultCompactSelection# > > > > > > > > testStuckStoreCompa > > > > > > > > > > >> ction > > > > > > > > > > >> > > >> > Thread=3D11 (was 7) > > > > > > > > > > >> > > >> > Potentially hanging thread: Monitor thread > for > > > > > > > TaskMonitor > > > > > > > > > > >> > > >> > java.lang.Thread.sleep(Native Metho= d) > > > > > > > > > > >> > > >> > org.apache.hadoop.hbase. > > > > > > monitoring.TaskMonitor$ > > > > > > > > > > >> > > >> > MonitorRunnable.run(TaskMonitor.java:302) > > > > > > > > > > >> > > >> > java.lang.Thread.run(Thread. > java:748) > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > Potentially hanging thread: > > org.apache.hadoop.fs. > > > > > > > > > > >> > > FileSystem$Statistics$ > > > > > > > > > > >> > > >> > StatisticsDataReferenceCleaner > > > > > > > > > > >> > > >> > java.lang.Object.wait(Native Method= ) > > > > > > > > > > >> > > >> > java.lang.ref.ReferenceQueue. > > > > > > > > > > remove(ReferenceQueue.java: > > > > > > > > > > >> > 143) > > > > > > > > > > >> > > >> > java.lang.ref.ReferenceQueue. > > > > > > > > > > remove(ReferenceQueue.java: > > > > > > > > > > >> > 164) > > > > > > > > > > >> > > >> > org.apache.hadoop.fs. > > > > FileSystem$Statistics$ > > > > > > > > > > >> > > >> > StatisticsDataReferenceCleaner > > > > > > .run(FileSystem.java:3063) > > > > > > > > > > >> > > >> > java.lang.Thread.run(Thread. > java:748) > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > Potentially hanging thread: > > > > > LruBlockCacheStatsExecutor > > > > > > > > > > >> > > >> > sun.misc.Unsafe.park(Native Method) > > > > > > > > > > >> > > >> > java.util.concurrent.locks. > > > > > > > LockSupport.parkNanos( > > > > > > > > > > >> > > >> > LockSupport.java:215) > > > > > > > > > > >> > > >> > java.util.concurrent.locks. > > > > > > > > > AbstractQueuedSynchronizer$ > > > > > > > > > > >> > > >> > ConditionObject.awaitNanos( > > > > > AbstractQueuedSynchronizer. > > > > > > > > > > java:2078) > > > > > > > > > > >> > > >> > java.util.concurrent. > > > > > > > ScheduledThreadPoolExecutor$ > > > > > > > > > > >> > > >> > DelayedWorkQueue.take( > > > ScheduledThreadPoolExecutor. > > > > > > > > java:1093) > > > > > > > > > > >> > > >> > java.util.concurrent. > > > > > > > ScheduledThreadPoolExecutor$ > > > > > > > > > > >> > > >> > DelayedWorkQueue.take( > > > ScheduledThreadPoolExecutor. > > > > > > > > java:809) > > > > > > > > > > >> > > >> > java.util.concurrent. > > > > > > ThreadPoolExecutor.getTask( > > > > > > > > > > >> > > >> > ThreadPoolExecutor.java:1074) > > > > > > > > > > >> > > >> > java.util.concurrent. > > > > > > > ThreadPoolExecutor.runWorker( > > > > > > > > > > >> > > >> > ThreadPoolExecutor.java:1134) > > > > > > > > > > >> > > >> > java.util.concurrent. > > > > > > > > ThreadPoolExecutor$Worker.run( > > > > > > > > > > >> > > >> > ThreadPoolExecutor.java:624) > > > > > > > > > > >> > > >> > java.lang.Thread.run(Thread. > java:748) > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > Potentially hanging thread: StoreOpener- > > > > > > > > > > >> > > 22ce1d683ba4b6b9373a3c541ebab2 > > > > > > > > > > >> > > >> > a2-1.LruBlockCache.EvictionThread > > > > > > > > > > >> > > >> > java.lang.Object.wait(Native Method= ) > > > > > > > > > > >> > > >> > org.apache.hadoop.hbase.io.hf > > > > > > > > > > >> ile.LruBlockCache$EvictionThre > > > > > > > > > > >> > > >> ad.run( > > > > > > > > > > >> > > >> > LruBlockCache.java:894) > > > > > > > > > > >> > > >> > java.lang.Thread.run(Thread. > java:748) > > > > > > > > > > >> > > >> > - Thread LEAK? -, OpenFileDescriptor=3D232 > (was > > > > 232), > > > > > > > > > > >> > > >> > MaxFileDescriptor=3D1048576 (was 1048576), > > > > > > > > > SystemLoadAverage=3D204 > > > > > > > > > > >> (was > > > > > > > > > > >> > > >> 204), > > > > > > > > > > >> > > >> > ProcessCount=3D273 (was 273), > > > AvailableMemoryMB=3D4049 > > > > > (was > > > > > > > > 4132) > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > Full log attached > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > Thanks, > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > JMS > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > 2018-01-06 9:34 GMT-05:00 Mike Drob < > > > > > mdrob@apache.org > > > > > > >: > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> >> I can reproduce the issue locally. I think > it > > > has > > > > to > > > > > > do > > > > > > > > with > > > > > > > > > > the > > > > > > > > > > >> > java > > > > > > > > > > >> > > >> >> version being used - IIRC this is related = to > > the > > > > > > version > > > > > > > > of > > > > > > > > > > java > > > > > > > > > > >> > > used, > > > > > > > > > > >> > > >> but > > > > > > > > > > >> > > >> >> we can discuss in more detail on the JIRA. > > > > > > > > > > >> > > >> >> > > > > > > > > > > >> > > >> >> https://issues.apache.org/ > > > jira/browse/HBASE-19721 > > > > > > > > > > >> > > >> >> > > > > > > > > > > >> > > >> >> Thanks, JMS! > > > > > > > > > > >> > > >> >> > > > > > > > > > > >> > > >> >> On Sat, Jan 6, 2018 at 6:42 AM, Jean-Marc > > > > Spaggiari > > > > > < > > > > > > > > > > >> > > >> >> jean-marc@spaggiari.org > > > > > > > > > > >> > > >> >> > wrote: > > > > > > > > > > >> > > >> >> > > > > > > > > > > >> > > >> >> > How you guys are able to get the tests > > > running? > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > For me it keeps failing on > > > > > > > TestReversedScannerCallable. > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > I tried many times, always fails in the > same > > > > > place. > > > > > > > I'm > > > > > > > > > > >> running > > > > > > > > > > >> > on > > > > > > > > > > >> > > a > > > > > > > > > > >> > > >> 4GB > > > > > > > > > > >> > > >> >> > tmpfs. Details are below. Am I doing > > something > > > > > > wrong? > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > JM > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > ./dev-support/hbasetests.sh runAllTests > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > [INFO] Running org.apache.hadoop.hbase. > > > > > > > > > client.TestOperation > > > > > > > > > > >> > > >> >> > [INFO] > > > > > > > > > > >> > > >> >> > [INFO] Results: > > > > > > > > > > >> > > >> >> > [INFO] > > > > > > > > > > >> > > >> >> > [ERROR] Errors: > > > > > > > > > > >> > > >> >> > [ERROR] TestReversedScannerCallable. > > > > unnecessary > > > > > > > > Mockito > > > > > > > > > > >> > > stubbings > > > > > > > > > > >> > > >> =C2=BB > > > > > > > > > > >> > > >> >> > UnnecessaryStubbing > > > > > > > > > > >> > > >> >> > [INFO] > > > > > > > > > > >> > > >> >> > [ERROR] Tests run: 245, Failures: 0, > Errors: > > > 1, > > > > > > > > Skipped: 8 > > > > > > > > > > >> > > >> >> > [INFO] > > > > > > > > > > >> > > >> >> > [INFO] > > > > > > > > > > >> > > >> >> > ------------------------------ > > > > > > > > > > ------------------------------ > > > > > > > > > > >> > > >> >> ------------ > > > > > > > > > > >> > > >> >> > [INFO] Reactor Summary: > > > > > > > > > > >> > > >> >> > [INFO] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase > > > > .............................. > > > > > > > > > ......... > > > > > > > > > > >> > > SUCCESS > > > > > > > > > > >> > > >> [ > > > > > > > > > > >> > > >> >> > 1.409 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Checkstyle > > > > > > > > > .......................... > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 1.295 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Build Support > > > > > > > > > ....................... > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 0.038 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Error Prone Rules > > > > > > > > > ................... > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 1.069 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Annotations > > > > > > > > > ......................... > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 1.450 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Build Configuratio= n > > > > > > > > > ................. > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 0.073 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Shaded Protocol > > > > > > > > > ..................... > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 14.292 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Common > > > > > > > > > .............................. > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > >> >> [01:51 > > > > > > > > > > >> > > >> >> > min] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Metrics API > > > > > > > > > ......................... > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 2.878 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Hadoop Compatibili= ty > > > > > > > > > ................ > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 12.216 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Metrics > Implementation > > > > > > > > > .............. > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 7.206 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Hadoop Two > > Compatibility > > > > > > > > > ............ > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 12.440 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Protocol > > > > > > > > > ............................ > > > > > > > > > > >> > SUCCESS > > > > > > > > > > >> > > [ > > > > > > > > > > >> > > >> >> > 0.074 s] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Client > > > > > > > > > .............................. > > > > > > > > > > >> > FAILURE > > > > > > > > > > >> > > >> >> [02:10 > > > > > > > > > > >> > > >> >> > min] > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Zookeeper > > > > > > > > > ........................... > > > > > > > > > > >> > SKIPPED > > > > > > > > > > >> > > >> >> > [INFO] Apache HBase - Replication > > > > > > > > > ......................... > > > > > > > > > > >> > SKIPPED > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > ------------------------------ > > > > > > > > > > ------------------------------ > > > > > > > > > > >> > > >> >> > ------------------- > > > > > > > > > > >> > > >> >> > Test set: org.apache.hadoop.hbase.client= . > > > > > > > > > > >> > > TestReversedScannerCallable > > > > > > > > > > >> > > >> >> > ------------------------------ > > > > > > > > > > ------------------------------ > > > > > > > > > > >> > > >> >> > ------------------- > > > > > > > > > > >> > > >> >> > Tests run: 3, Failures: 0, Errors: 1, > > Skipped: > > > > 0, > > > > > > Time > > > > > > > > > > >> elapsed: > > > > > > > > > > >> > > >> 1.515 s > > > > > > > > > > >> > > >> >> <<< > > > > > > > > > > >> > > >> >> > FAILURE! - in > org.apache.hadoop.hbase.client > > > > > > > > > > >> > > >> >> .TestReversedScannerCallable > > > > > > > > > > >> > > >> >> > unnecessary Mockito > > > > > > > > > > >> > > >> >> > stubbings(org.apache.hadoop.hbase.client= . > > > > > > > > > > TestReversedScanner > > > > > > > > > > >> > > >> Callable) > > > > > > > > > > >> > > >> >> > Time > > > > > > > > > > >> > > >> >> > elapsed: 0.014 s <<< ERROR! > > > > > > > > > > >> > > >> >> > org.mockito.exceptions.misusing. > > > > > > > > > > UnnecessaryStubbingException: > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > Unnecessary stubbings detected in test > > class: > > > > > > > > > > >> > > >> >> TestReversedScannerCallable > > > > > > > > > > >> > > >> >> > Clean & maintainable test code requires > zero > > > > > > > unnecessary > > > > > > > > > > code. > > > > > > > > > > >> > > >> >> > Following stubbings are unnecessary (cli= ck > > to > > > > > > navigate > > > > > > > > to > > > > > > > > > > >> > relevant > > > > > > > > > > >> > > >> line > > > > > > > > > > >> > > >> >> of > > > > > > > > > > >> > > >> >> > code): > > > > > > > > > > >> > > >> >> > 1. -> at > > > > > > > > > > >> > > >> >> > org.apache.hadoop.hbase.client. > > > > > > > > > TestReversedScannerCallable. > > > > > > > > > > >> > setUp( > > > > > > > > > > >> > > >> >> > TestReversedScannerCallable.java:66) > > > > > > > > > > >> > > >> >> > 2. -> at > > > > > > > > > > >> > > >> >> > org.apache.hadoop.hbase.client. > > > > > > > > > TestReversedScannerCallable. > > > > > > > > > > >> > setUp( > > > > > > > > > > >> > > >> >> > TestReversedScannerCallable.java:68) > > > > > > > > > > >> > > >> >> > Please remove unnecessary stubbings. Mor= e > > > info: > > > > > > > javadoc > > > > > > > > > for > > > > > > > > > > >> > > >> >> > UnnecessaryStubbingException class. > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > 2018-01-06 0:44 GMT-05:00 stack < > > > > > > saint.ack@gmail.com > > > > > > > >: > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > On Jan 5, 2018 4:44 PM, "Apekshit > Sharma" > > < > > > > > > > > > > >> appy@cloudera.com> > > > > > > > > > > >> > > >> wrote: > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > bq. Care needs to be exercised > > backporting. > > > > Bug > > > > > > > fixes > > > > > > > > > only > > > > > > > > > > >> > > please. > > > > > > > > > > >> > > >> If > > > > > > > > > > >> > > >> >> in > > > > > > > > > > >> > > >> >> > > doubt, ping me, the RM, please. Thanks= . > > > > > > > > > > >> > > >> >> > > In that case, shouldn't we branch out > > > > > branch-2.0? > > > > > > We > > > > > > > > can > > > > > > > > > > >> then > > > > > > > > > > >> > do > > > > > > > > > > >> > > >> >> normal > > > > > > > > > > >> > > >> >> > > backports to branch-2 and only bug fix= es > > to > > > > > > > > branch-2.0. > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > Don't you think we have enough branche= s > > > > already > > > > > > > mighty > > > > > > > > > > Appy? > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > No new features on branch-2? New > features > > > are > > > > in > > > > > > > > > > >> master/3.0.0 > > > > > > > > > > >> > > only? > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > S > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > On Fri, Jan 5, 2018 at 9:48 AM, Andrew > > > > Purtell < > > > > > > > > > > >> > > >> apurtell@apache.org> > > > > > > > > > > >> > > >> >> > > wrote: > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > TestMemstoreLABWithoutPool is a flak= e, > > > not a > > > > > > > > > consistent > > > > > > > > > > >> fail. > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > >> > > >> >> > > > On Fri, Jan 5, 2018 at 7:18 AM, Stac= k > < > > > > > > > > > stack@duboce.net > > > > > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > >> > > >> >> > > > > On Thu, Jan 4, 2018 at 2:24 PM, > Andrew > > > > > > Purtell < > > > > > > > > > > >> > > >> >> apurtell@apache.org> > > > > > > > > > > >> > > >> >> > > > > wrote: > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > This one is probably my fault: > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > TestDefaultCompactSelection > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > HBASE-19406 > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > Balazs fixed it above, HBASE-19666 > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > It can easily be reverted. The > > failure > > > > of > > > > > > > > interest > > > > > > > > > > >> > > >> >> > > > > > is TestMemstoreLABWithoutPool.te= s > > > > > > > > > > >> > > >> tLABChunkQueueWithMultipleMSLA > > > > > > > > > > >> > > >> >> Bs. > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > This seems fine. Passes in nightly > > > > > > > > > > >> > > >> >> > > > > https://builds.apache.org/view > > > > > > > > > > >> /H-L/view/HBase/job/HBase% > > > > > > > > > > >> > > >> >> > > > > 20Nightly/job/branch-2/171/tes > > > > > > > > > > >> tReport/org.apache.hadoop. > > > > > > > > > > >> > > >> >> > > > > hbase.regionserver/ > > > > > > TestMemstoreLABWithoutPool/ > > > > > > > > > > >> > > >> >> > > > > and locally against the tag. It > fails > > > > > > > consistently > > > > > > > > > for > > > > > > > > > > >> you > > > > > > > > > > >> > > >> Andrew? > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > > Should all unit tests pass on = a > > > beta? > > > > I > > > > > > > think > > > > > > > > > so, > > > > > > > > > > at > > > > > > > > > > >> > > least > > > > > > > > > > >> > > >> if > > > > > > > > > > >> > > >> >> the > > > > > > > > > > >> > > >> >> > > > > > failures > > > > > > > > > > >> > > >> >> > > > > > > are 100% repeatable. > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > This is fair. Let me squash this R= C > > and > > > > roll > > > > > > > > > another. > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > Will put it up in a few hours. > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > Thanks, > > > > > > > > > > >> > > >> >> > > > > S > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > > -0 > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > Checked sums and signatures: o= k > > > > > > > > > > >> > > >> >> > > > > > > RAT check: ok > > > > > > > > > > >> > > >> >> > > > > > > Built from source: ok (8u144) > > > > > > > > > > >> > > >> >> > > > > > > Ran unit tests: some failures > > > (8u144) > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > [ERROR] > > > > TestDefaultCompactSelection.t > > > > > > > > > > >> > > >> >> estCompactionRatio:74-> > > > > > > > > > > >> > > >> >> > > > > > > TestCompactionPolicy.compactEq > > > > > > > > > > >> > > >> uals:182->TestCompactionPolicy. > > > > > > > > > > >> > > >> >> > > > > > compactEquals:201 > > > > > > > > > > >> > > >> >> > > > > > > expected:<[[4, 2, 1]]> but > > > was:<[[]]> > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > [ERROR] > > > TestDefaultCompactSelection. > > > > > > > > > > >> > > >> >> > > testStuckStoreCompaction:145-> > > > > > > > > > > >> > > >> >> > > > > > > TestCompactionPolicy.compactEq > > > > > > > > > > >> > > >> uals:182->TestCompactionPolicy. > > > > > > > > > > >> > > >> >> > > > > > compactEquals:201 > > > > > > > > > > >> > > >> >> > > > > > > expected:<[[]30, 30, 30]> but > > > > was:<[[99, > > > > > > 30, > > > > > > > > > ]30, > > > > > > > > > > >> 30, > > > > > > > > > > >> > > 30]> > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > [ERROR] > > > TestMemstoreLABWithoutPool. > > > > > > > > > > >> > > >> >> > > testLABChunkQueueWithMultipleM > > > > > > > > > > >> > > >> >> > > > > > SLABs:143 > > > > > > > > > > >> > > >> >> > > > > > > All the chunks must have been > > > cleared > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > On Fri, Dec 29, 2017 at 10:15 > AM, > > > > Stack > > > > > < > > > > > > > > > > >> > > stack@duboce.net> > > > > > > > > > > >> > > >> >> > wrote: > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > >> The first release candidate f= or > > > HBase > > > > > > > > > > 2.0.0-beta-1 > > > > > > > > > > >> is > > > > > > > > > > >> > up > > > > > > > > > > >> > > >> at: > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> > https://dist.apache.org/repos/ > > > > > > > > > > >> > > >> dist/dev/hbase/hbase-2.0.0- > > > > > > > > > > >> > > >> >> > > > beta-1-RC0/ > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> Maven artifacts are available > > from > > > a > > > > > > > staging > > > > > > > > > > >> directory > > > > > > > > > > >> > > >> here: > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> > https://repository.apache.org/ > > > > > > > > > > >> content/repositories/ > > > > > > > > > > >> > > >> >> > > > > orgapachehbase-1188 > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> All was signed with my key at > > > > 8ACC93D2 > > > > > > [1] > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> I tagged the RC as > > 2.0.0-beta-1-RC0 > > > > > > > > > > >> > > >> >> > > > > > >> ( > 0907563eb72697b394b8b960fe5488 > > > > > > 7d6ff304fd) > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> hbase-2.0.0-beta-1 is our fir= st > > > beta > > > > > > > release. > > > > > > > > > It > > > > > > > > > > >> > > includes > > > > > > > > > > >> > > >> all > > > > > > > > > > >> > > >> >> > that > > > > > > > > > > >> > > >> >> > > > was > > > > > > > > > > >> > > >> >> > > > > > in > > > > > > > > > > >> > > >> >> > > > > > >> previous alphas (new assignme= nt > > > > > manager, > > > > > > > > > offheap > > > > > > > > > > >> > > >> read/write > > > > > > > > > > >> > > >> >> > path, > > > > > > > > > > >> > > >> >> > > > > > >> in-memory > > > > > > > > > > >> > > >> >> > > > > > >> compactions, etc.). The APIs > and > > > > > > > feature-set > > > > > > > > > are > > > > > > > > > > >> > sealed. > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> hbase-2.0.0-beta-1 is a > > > > > > not-for-production > > > > > > > > > > preview > > > > > > > > > > >> of > > > > > > > > > > >> > > >> >> > hbase-2.0.0. > > > > > > > > > > >> > > >> >> > > > It > > > > > > > > > > >> > > >> >> > > > > is > > > > > > > > > > >> > > >> >> > > > > > >> meant for devs and > downstreamers > > to > > > > > test > > > > > > > > drive > > > > > > > > > > and > > > > > > > > > > >> > flag > > > > > > > > > > >> > > us > > > > > > > > > > >> > > >> >> if we > > > > > > > > > > >> > > >> >> > > > > messed > > > > > > > > > > >> > > >> >> > > > > > up > > > > > > > > > > >> > > >> >> > > > > > >> on anything ahead of our > rolling > > > GAs. > > > > > We > > > > > > > are > > > > > > > > > > >> > particular > > > > > > > > > > >> > > >> >> > interested > > > > > > > > > > >> > > >> >> > > > in > > > > > > > > > > >> > > >> >> > > > > > >> hearing from Coprocessor > > > developers. > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> The list of features addresse= d > in > > > > 2.0.0 > > > > > > so > > > > > > > > far > > > > > > > > > > can > > > > > > > > > > >> be > > > > > > > > > > >> > > >> found > > > > > > > > > > >> > > >> >> here > > > > > > > > > > >> > > >> >> > > > [3]. > > > > > > > > > > >> > > >> >> > > > > > >> There > > > > > > > > > > >> > > >> >> > > > > > >> are thousands. The list of ~2= k+ > > > fixes > > > > > in > > > > > > > > 2.0.0 > > > > > > > > > > >> > > exclusively > > > > > > > > > > >> > > >> >> can > > > > > > > > > > >> > > >> >> > be > > > > > > > > > > >> > > >> >> > > > > found > > > > > > > > > > >> > > >> >> > > > > > >> here [4] (My JIRA JQL foo is = a > > bit > > > > > dodgy > > > > > > -- > > > > > > > > > > >> forgive me > > > > > > > > > > >> > > if > > > > > > > > > > >> > > >> >> > > mistakes). > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> I've updated our overview doc= . > on > > > the > > > > > > state > > > > > > > > of > > > > > > > > > > >> 2.0.0 > > > > > > > > > > >> > > [6]. > > > > > > > > > > >> > > >> >> We'll > > > > > > > > > > >> > > >> >> > do > > > > > > > > > > >> > > >> >> > > > one > > > > > > > > > > >> > > >> >> > > > > > >> more > > > > > > > > > > >> > > >> >> > > > > > >> beta before we put up our fir= st > > > 2.0.0 > > > > > > > Release > > > > > > > > > > >> > Candidate > > > > > > > > > > >> > > by > > > > > > > > > > >> > > >> >> the > > > > > > > > > > >> > > >> >> > end > > > > > > > > > > >> > > >> >> > > > of > > > > > > > > > > >> > > >> >> > > > > > >> January, 2.0.0-beta-2. Its > focus > > > will > > > > > be > > > > > > > > making > > > > > > > > > > it > > > > > > > > > > >> so > > > > > > > > > > >> > > >> users > > > > > > > > > > >> > > >> >> can > > > > > > > > > > >> > > >> >> > do > > > > > > > > > > >> > > >> >> > > a > > > > > > > > > > >> > > >> >> > > > > > >> rolling upgrade on to hbase-2= .x > > > from > > > > > > > > hbase-1.x > > > > > > > > > > (and > > > > > > > > > > >> > any > > > > > > > > > > >> > > >> bug > > > > > > > > > > >> > > >> >> > fixes > > > > > > > > > > >> > > >> >> > > > > found > > > > > > > > > > >> > > >> >> > > > > > >> running beta-1). Here is the > list > > > of > > > > > what > > > > > > > we > > > > > > > > > have > > > > > > > > > > >> > > >> targeted so > > > > > > > > > > >> > > >> >> > far > > > > > > > > > > >> > > >> >> > > > for > > > > > > > > > > >> > > >> >> > > > > > >> beta-2 [5]. Check it out. > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> One knownissue is that the Us= er > > API > > > > has > > > > > > not > > > > > > > > > been > > > > > > > > > > >> > > properly > > > > > > > > > > >> > > >> >> > filtered > > > > > > > > > > >> > > >> >> > > > so > > > > > > > > > > >> > > >> >> > > > > it > > > > > > > > > > >> > > >> >> > > > > > >> shows more than just > > > > InterfaceAudience > > > > > > > Public > > > > > > > > > > >> content > > > > > > > > > > >> > > >> >> > > (HBASE-19663, > > > > > > > > > > >> > > >> >> > > > to > > > > > > > > > > >> > > >> >> > > > > > be > > > > > > > > > > >> > > >> >> > > > > > >> fixed by beta-2). > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> Please take this beta for a > spin. > > > > > Please > > > > > > > vote > > > > > > > > > on > > > > > > > > > > >> > whether > > > > > > > > > > >> > > >> it > > > > > > > > > > >> > > >> >> ok > > > > > > > > > > >> > > >> >> > to > > > > > > > > > > >> > > >> >> > > > put > > > > > > > > > > >> > > >> >> > > > > > out > > > > > > > > > > >> > > >> >> > > > > > >> this RC as our first beta (No= te > > > > CHANGES > > > > > > has > > > > > > > > not > > > > > > > > > > yet > > > > > > > > > > >> > been > > > > > > > > > > >> > > >> >> > updated). > > > > > > > > > > >> > > >> >> > > > Let > > > > > > > > > > >> > > >> >> > > > > > the > > > > > > > > > > >> > > >> >> > > > > > >> VOTE be open for 72 hours > > (Monday) > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> Thanks, > > > > > > > > > > >> > > >> >> > > > > > >> Your 2.0.0 Release Manager > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > >> 1. > http://pgp.mit.edu/pks/lookup > > ? > > > > > > > > > op=3Dget&search=3D > > > > > > > > > > >> > > >> >> > 0x9816C7FC8ACC93D2 > > > > > > > > > > >> > > >> >> > > > > > >> 3. https://goo.gl/scYjJr > > > > > > > > > > >> > > >> >> > > > > > >> 4. https://goo.gl/dFFT8b > > > > > > > > > > >> > > >> >> > > > > > >> 5. > > https://issues.apache.org/jira > > > > > > > > > > >> > > >> /projects/HBASE/versions/ > > > > > > > > > > >> > > >> >> > > 12340862 > > > > > > > > > > >> > > >> >> > > > > > >> 6. https://docs.google.com/ > > > > document/d/ > > > > > > > > > > >> > > >> >> > > 1WCsVlnHjJeKUcl7wHwqb4z9iEu_ > > > > > > > > > > >> > > >> >> > > > > > >> ktczrlKHK8N4SZzs/ > > > > > > > > > > >> > > >> >> > > > > > >> > > document/d/ > > > > > > > > > > >> > > >> >> > 1WCsVlnHjJeKUcl7wHwqb4z9iEu_ > > > > > > > > > > >> > > >> >> > > > > > ktczrlKHK8N4SZzs/> > > > > > > > > > > >> > > >> >> > > > > > >> > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > -- > > > > > > > > > > >> > > >> >> > > > > > > Best regards, > > > > > > > > > > >> > > >> >> > > > > > > Andrew > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > Words like orphans lost among > the > > > > > > crosstalk, > > > > > > > > > > meaning > > > > > > > > > > >> > torn > > > > > > > > > > >> > > >> from > > > > > > > > > > >> > > >> >> > > > truth's > > > > > > > > > > >> > > >> >> > > > > > > decrepit hands > > > > > > > > > > >> > > >> >> > > > > > > - A23, Crosstalk > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > -- > > > > > > > > > > >> > > >> >> > > > > > Best regards, > > > > > > > > > > >> > > >> >> > > > > > Andrew > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > Words like orphans lost among th= e > > > > > crosstalk, > > > > > > > > > meaning > > > > > > > > > > >> torn > > > > > > > > > > >> > > >> from > > > > > > > > > > >> > > >> >> > > truth's > > > > > > > > > > >> > > >> >> > > > > > decrepit hands > > > > > > > > > > >> > > >> >> > > > > > - A23, Crosstalk > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > >> > > >> >> > > > -- > > > > > > > > > > >> > > >> >> > > > Best regards, > > > > > > > > > > >> > > >> >> > > > Andrew > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > >> > > >> >> > > > Words like orphans lost among the > > > crosstalk, > > > > > > > meaning > > > > > > > > > > torn > > > > > > > > > > >> > from > > > > > > > > > > >> > > >> >> truth's > > > > > > > > > > >> > > >> >> > > > decrepit hands > > > > > > > > > > >> > > >> >> > > > - A23, Crosstalk > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > -- > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > -- Appy > > > > > > > > > > >> > > >> >> > > > > > > > > > > > > >> > > >> >> > > > > > > > > > > > >> > > >> >> > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > -- > > > > > > > > > > >> > > > > > > > > > > > >> > -- Appy > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Best regards, > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > Words like orphans lost among the crosstalk, meaning torn > > from > > > > > > truth's > > > > > > > > > decrepit hands > > > > > > > > > - A23, Crosstalk > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --=20 -- Appy --001a1144402a7254700562770a39--