Mailing-List: contact dev-help@tephra.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@tephra.incubator.apache.org
Subject: Re: Tephra unreliable in CI environments
To: dev@tephra.incubator.apache.org
References: <564cf774-cb6d-5a71-2bb2-e2488745d0a2@gmail.com>
 <a97c3b26-38c8-a389-94be-336d4e006367@gmail.com>
 <CAC9o21QDg=tsYD8civt=StnOJVVdSMnrB1Sr8Mx3OvZ6KRybbA@mail.gmail.com>
 <10542db7-26fc-83cd-8126-9a34644bc484@gmail.com>
 <CAC9o21RRjkREg6JLNmu-+-G5TcXWwr3WWbJgDmJ51Ju98hLp4w@mail.gmail.com>
From: F21 <f21.groups@gmail.com>
Message-ID: <ca21f3f0-2e13-2466-9c6a-1b69a715fd09@gmail.com>
Date: Wed, 28 Sep 2016 13:57:46 +1000
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.3.0
MIME-Version: 1.0
In-Reply-To: <CAC9o21RRjkREg6JLNmu-+-G5TcXWwr3WWbJgDmJ51Ju98hLp4w@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Wed, 28 Sep 2016 03:58:06 -0000

Hi Poorna,

Thanks, that narrows down the problem. I was spinning up a few VMs with 
various versions of Ubuntu and their kernels, but that didn't shed any 
light on the problem.

You mentioned that adding the guava-13 jar to the classpath before the 
guava-12 jar would be a workaround. I am using Phoenix 4.8.1-rc0, so 
what would be the best way to do this?

Cheers,
Francis

On 28/09/2016 1:43 PM, Poorna Chandra wrote:
> Hi Francis,
>
> This is due to guava-12 vs guava-13 incompatibility [1]. Tephra depends on
> guava-13 and HBase depends on guava-12. Depending on how the OS orders the
> jars in the classpath, sometimes guava-12 may appear earlier in the
> classpath. This leads to the NoSuchMethodError exception. We are planning
> on removing Tephra's dependency on guava-13 in the next release. Until then
> a workaround is to add guava-13 jar into the classpath before guava-12 jar.
>
> Thanks,
> Poorna.
>
> [1] - https://issues.apache.org/jira/browse/TEPHRA-181
>
>
> On Tue, Sep 27, 2016 at 10:09 PM, F21 <f21.groups@gmail.com> wrote:
>
>> Hi Poorna,
>>
>> That would be very helpful! Unfortunately, I ran into the same issue where
>> the image no longer works correct on my dev environment, but works properly
>> on travis.
>>
>> I am not receiving this error:
>>
>> java.lang.NoSuchMethodError:
>> co.cask.tephra.TransactionManager.addListener(Lcom/google/co
>> mmon/util/concurrent/Service$Listener;Ljava/util/concurrent/Executor;)V
>>           at
>> co.cask.tephra.distributed.TransactionService$1.leader(Trans
>> actionService.java:83)
>>           at
>> org.apache.twill.internal.zookeeper.LeaderElection.becomeLea
>> der(LeaderElection.java:229)
>>           at
>> org.apache.twill.internal.zookeeper.LeaderElection.access$
>> 1800(LeaderElection.java:53)
>>           at
>> org.apache.twill.internal.zookeeper.LeaderElection$5.onSucce
>> ss(LeaderElection.java:207)
>>           at
>> org.apache.twill.internal.zookeeper.LeaderElection$5.onSucce
>> ss(LeaderElection.java:186)
>>           at
>> com.google.common.util.concurrent.Futures$5.run(Futures.java:768)
>>           at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>>           at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>>           at java.lang.Thread.run(Thread.java:745)
>>
>> This is the exact same problem I encountered and posted on the list about
>> previously: https://mail-archives.apache.org/mod_mbox/phoenix-user/20160
>> 3.mbox/%3C56FC727E.30906@gmail.com%3E
>>
>> It's puzzling that an identical image will behave differently on different
>> systems. Does tephra use any kernel functionalities directly?
>>
>> Cheers,
>> Francis
>>
>>
>> On 28/09/2016 12:04 PM, Poorna Chandra wrote:
>>
>>> Hi Francis,
>>>
>>> Tephra startup script redirects all logs to a file by default. To help
>>> debug such issues, you could update the Tephra startup script [1] to log
>>> everything to stdout instead of a file when running inside docker. We are
>>> also planning on adding a --foreground option to Tephra startup script, in
>>> which case the logs will be written to stdout directly. This will help in
>>> debugging in future.
>>>
>>> Thanks,
>>> Poorna.
>>>
>>> [1] - https://github.com/apache/incubator-tephra/blob/master/bin/
>>> tephra#L175
>>>
>>>
>>> On Tue, Sep 27, 2016 at 12:59 AM, F21 <f21.groups@gmail.com> wrote:
>>>
>>> I was able to get it to run reliably with Phoenix 4.8.1-rc0. Another part
>>>> of the equation was forcing travis to use their Ubuntu 14.04 environment
>>>> rather than the default 12.04 environment. I am assuming 12.04 had an
>>>> older
>>>> kernel which prevent docker images from working correctly.
>>>>
>>>> On 27/09/2016 10:15 AM, F21 wrote:
>>>>
>>>> Hi all,
>>>>> I have created a docker image containing HBase 1.2.3 and Phoenix 4.8.0.
>>>>> See: https://github.com/Boostport/hbase-phoenix-all-in-one
>>>>>
>>>>> When running tests against the image on my machine, tephra works
>>>>> perfectly.
>>>>>
>>>>> However, tephra seems to be unreliable in CI environments. It seems that
>>>>> the tx service is not discovered:
>>>>>
>>>>> RuntimeException: java.lang.Exception: Thrift error for
>>>>> org.apache.tephra.distributed.TransactionServiceClient$2@3e9e291e:
>>>>> Unable to discover tx service. -> Exception: Thrift error for
>>>>> org.apache.tephra.distributed.TransactionServiceClient$2@3e9e291e:
>>>>> Unable to discover tx service. -> TException: Unable to discover tx
>>>>> service.
>>>>>
>>>>> Here's a build on wercker which shows tephra failing:
>>>>> https://app.wercker.com/boostport/avatica/runs/build/57e9b5d
>>>>> 170a35501008402b4?step=57e9b5f72c15ad000127a534
>>>>>
>>>>> I also tried travis, but the same issue occurs:
>>>>> https://travis-ci.org/Boostport/avatica/builds/162952367
>>>>>
>>>>> Since I am unable to ssh into those docker container on wercker or
>>>>> travis, it is hard to debug what's causing tephra to fail. I am hoping
>>>>> that
>>>>> the issue is related to TEPHRA-179 (and a few other JIRAs related to it)
>>>>> which I reported a few weeks ago and has since been fixed.
>>>>>
>>>>> Has anyone else ran into similar problems? I would love to hear your
>>>>> thoughts.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Francis
>>>>>
>>>>>
>>>>>