tephra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Poorna Chandra <poornachan...@gmail.com>
Subject Re: Tephra unreliable in CI environments
Date Tue, 11 Oct 2016 04:43:45 GMT
Cool, thanks for the update!

Poorna.

On Oct 10, 2016 9:34 PM, "F21" <f21.groups@gmail.com> wrote:

> Hi Poorna,
>
> I was able to resolve this with James' help in PHOENIX-3259 by deleting
> the guava jar in the HBase distribution and replacing it with the guava-13
> jar.
>
> Thanks,
> Francis
>
> On 11/10/2016 3:09 PM, Poorna Chandra wrote:
>
>> Hi Francis,
>>
>> (Apology for the delayed response, I was out of office for the past few
>> weeks.)
>>
>> Do you see a guava-13 jar in the classpath of Tephra? It should be in
>> <tephra-home>/lib directory. You can define environment variable CLASSPATH
>> pointing to path of guava-13 jar before starting Tephra. This will put
>> guava-13 jar earlier in Tephra's classpath.
>>
>> Thanks,
>> Poorna.
>>
>>
>> On Tue, Sep 27, 2016 at 8:57 PM, F21 <f21.groups@gmail.com> wrote:
>>
>> Hi Poorna,
>>>
>>> Thanks, that narrows down the problem. I was spinning up a few VMs with
>>> various versions of Ubuntu and their kernels, but that didn't shed any
>>> light on the problem.
>>>
>>> You mentioned that adding the guava-13 jar to the classpath before the
>>> guava-12 jar would be a workaround. I am using Phoenix 4.8.1-rc0, so what
>>> would be the best way to do this?
>>>
>>> Cheers,
>>> Francis
>>>
>>> On 28/09/2016 1:43 PM, Poorna Chandra wrote:
>>>
>>> Hi Francis,
>>>>
>>>> This is due to guava-12 vs guava-13 incompatibility [1]. Tephra depends
>>>> on
>>>> guava-13 and HBase depends on guava-12. Depending on how the OS orders
>>>> the
>>>> jars in the classpath, sometimes guava-12 may appear earlier in the
>>>> classpath. This leads to the NoSuchMethodError exception. We are
>>>> planning
>>>> on removing Tephra's dependency on guava-13 in the next release. Until
>>>> then
>>>> a workaround is to add guava-13 jar into the classpath before guava-12
>>>> jar.
>>>>
>>>> Thanks,
>>>> Poorna.
>>>>
>>>> [1] - https://issues.apache.org/jira/browse/TEPHRA-181
>>>>
>>>>
>>>> On Tue, Sep 27, 2016 at 10:09 PM, F21 <f21.groups@gmail.com> wrote:
>>>>
>>>> Hi Poorna,
>>>>
>>>>> That would be very helpful! Unfortunately, I ran into the same issue
>>>>> where
>>>>> the image no longer works correct on my dev environment, but works
>>>>> properly
>>>>> on travis.
>>>>>
>>>>> I am not receiving this error:
>>>>>
>>>>> java.lang.NoSuchMethodError:
>>>>> co.cask.tephra.TransactionManager.addListener(Lcom/google/co
>>>>> mmon/util/concurrent/Service$Listener;Ljava/util/concurrent/
>>>>> Executor;)V
>>>>>            at
>>>>> co.cask.tephra.distributed.TransactionService$1.leader(Trans
>>>>> actionService.java:83)
>>>>>            at
>>>>> org.apache.twill.internal.zookeeper.LeaderElection.becomeLea
>>>>> der(LeaderElection.java:229)
>>>>>            at
>>>>> org.apache.twill.internal.zookeeper.LeaderElection.access$
>>>>> 1800(LeaderElection.java:53)
>>>>>            at
>>>>> org.apache.twill.internal.zookeeper.LeaderElection$5.onSucce
>>>>> ss(LeaderElection.java:207)
>>>>>            at
>>>>> org.apache.twill.internal.zookeeper.LeaderElection$5.onSucce
>>>>> ss(LeaderElection.java:186)
>>>>>            at
>>>>> com.google.common.util.concurrent.Futures$5.run(Futures.java:768)
>>>>>            at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>>> Executor.java:1142)
>>>>>            at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>>> lExecutor.java:617)
>>>>>            at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> This is the exact same problem I encountered and posted on the list
>>>>> about
>>>>> previously: https://mail-archives.apache.o
>>>>> rg/mod_mbox/phoenix-user/20160
>>>>> 3.mbox/%3C56FC727E.30906@gmail.com%3E
>>>>>
>>>>> It's puzzling that an identical image will behave differently on
>>>>> different
>>>>> systems. Does tephra use any kernel functionalities directly?
>>>>>
>>>>> Cheers,
>>>>> Francis
>>>>>
>>>>>
>>>>> On 28/09/2016 12:04 PM, Poorna Chandra wrote:
>>>>>
>>>>> Hi Francis,
>>>>>
>>>>>> Tephra startup script redirects all logs to a file by default. To
help
>>>>>> debug such issues, you could update the Tephra startup script [1]
to
>>>>>> log
>>>>>> everything to stdout instead of a file when running inside docker.
We
>>>>>> are
>>>>>> also planning on adding a --foreground option to Tephra startup
>>>>>> script,
>>>>>> in
>>>>>> which case the logs will be written to stdout directly. This will
help
>>>>>> in
>>>>>> debugging in future.
>>>>>>
>>>>>> Thanks,
>>>>>> Poorna.
>>>>>>
>>>>>> [1] - https://github.com/apache/incubator-tephra/blob/master/bin/
>>>>>> tephra#L175
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 27, 2016 at 12:59 AM, F21 <f21.groups@gmail.com>
wrote:
>>>>>>
>>>>>> I was able to get it to run reliably with Phoenix 4.8.1-rc0. Another
>>>>>> part
>>>>>>
>>>>>> of the equation was forcing travis to use their Ubuntu 14.04
>>>>>>> environment
>>>>>>> rather than the default 12.04 environment. I am assuming 12.04
had an
>>>>>>> older
>>>>>>> kernel which prevent docker images from working correctly.
>>>>>>>
>>>>>>> On 27/09/2016 10:15 AM, F21 wrote:
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I have created a docker image containing HBase 1.2.3 and Phoenix
>>>>>>>> 4.8.0.
>>>>>>>> See: https://github.com/Boostport/hbase-phoenix-all-in-one
>>>>>>>>
>>>>>>>> When running tests against the image on my machine, tephra
works
>>>>>>>> perfectly.
>>>>>>>>
>>>>>>>> However, tephra seems to be unreliable in CI environments.
It seems
>>>>>>>> that
>>>>>>>> the tx service is not discovered:
>>>>>>>>
>>>>>>>> RuntimeException: java.lang.Exception: Thrift error for
>>>>>>>> org.apache.tephra.distributed.TransactionServiceClient$2@3e9e291e:
>>>>>>>> Unable to discover tx service. -> Exception: Thrift error
for
>>>>>>>> org.apache.tephra.distributed.TransactionServiceClient$2@3e9e291e:
>>>>>>>> Unable to discover tx service. -> TException: Unable to
discover tx
>>>>>>>> service.
>>>>>>>>
>>>>>>>> Here's a build on wercker which shows tephra failing:
>>>>>>>> https://app.wercker.com/boostport/avatica/runs/build/57e9b5d
>>>>>>>> 170a35501008402b4?step=57e9b5f72c15ad000127a534
>>>>>>>>
>>>>>>>> I also tried travis, but the same issue occurs:
>>>>>>>> https://travis-ci.org/Boostport/avatica/builds/162952367
>>>>>>>>
>>>>>>>> Since I am unable to ssh into those docker container on wercker
or
>>>>>>>> travis, it is hard to debug what's causing tephra to fail.
I am
>>>>>>>> hoping
>>>>>>>> that
>>>>>>>> the issue is related to TEPHRA-179 (and a few other JIRAs
related to
>>>>>>>> it)
>>>>>>>> which I reported a few weeks ago and has since been fixed.
>>>>>>>>
>>>>>>>> Has anyone else ran into similar problems? I would love to
hear your
>>>>>>>> thoughts.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Francis
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message