hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6414) ATSv2 tests fail due to guava version upgrade
Date Sat, 01 Apr 2017 12:25:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952196#comment-15952196

Steve Loughran commented on YARN-6414:

* test wise, it shows that actually the JAR update process needs to have more rigorous testing:
the patch should go in for all the subprojects so yetus tests them all. I'll keep an eye on
future changes there
* circular dependencies are generally considered "bad form" and shouldn't have happened. More
specifically, the use of the acronym "DAG" in the maven dependency graph declares that the
graph is "acyclic". Pulling in hbase as a dependency of hadoop is trying to do what spark/hive
have achieved: created a cycle. Is there anyway to unwind the cycle so that there is an ATSvb2
server module independent of the others, which pulls in both. That way, it can take in any
shaded guava libs from Hadoop & whatever HBase needs, while also allowing Hadoop to make
progress on the plan to drop jersey 1 and so avoid version issues there ( HADOOP-13332 )

BTW, Hadoop may export Guava 11 but it's coded to not use classes cut out of later versions
(e.g. guava dropping stopwatch (HADOOP-11032). Even so, the need to up the base guava version
is a serious need, not just for downestream projects, but even because things like Curator
which we pull in depends on later versions. Every upgrade of curator has problems related
to guava  HADOOP-11102 , HADOOP-11612

Side issue: In an ideal world. Guava would be backwards compatible, It isn't , and we get
to deal with the pain. Whatever we do, something breaks. Oh, and then there's protobuf.

> ATSv2 tests fail due to guava version upgrade
> ---------------------------------------------
>                 Key: YARN-6414
>                 URL: https://issues.apache.org/jira/browse/YARN-6414
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: 3.0.0-alpha3
>         Environment: Ubuntu 14.04 
> x86, ppc64le
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>            Reporter: Sonia Garudi
>            Assignee: Haibo Chen
>              Labels: ppc64le, x86_64
>         Attachments: YARN-6414.00.patch, YARN-6414.01.patch
> Test failures seen in Hadoop YARN Timeline Service HBase tests project with following
error :
> {code}
> java.lang.NoClassDefFoundError: com/google/common/io/LimitInputStream
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:223)
>         at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:913)
>         at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:899)
>         at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:722)
>         at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:660)
>         at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:279)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
>         at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529)
>         at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585)
>         at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:751)
>         at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:735)
>         at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407)
>         at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:998)
>         at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:869)
>         at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:704)
>         at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:642)
>         at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:590)
>         at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:987)
>         at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:868)
>         at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:862)
>         at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
>         at org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:88)
> {code}
> Below are the failing tests :
> {code}
> TestTimelineReaderWebServicesHBaseStorage.setup:88 » NoClassDefFound com/googl...
>   TestHBaseTimelineStorageEntities.setupBeforeClass:100 » NoClassDefFound com/go...
>   TestHBaseStorageFlowActivity.setupBeforeClass:72 » NoClassDefFound com/google/...
>   TestHBaseStorageFlowRunCompaction.setupBeforeClass:84 » NoClassDefFound com/go...
>   TestHBaseStorageFlowRun.setupBeforeClass:86 » NoClassDefFound com/google/commo...
>   TestHBaseTimelineStorageApps.setupBeforeClass:93 » NoClassDefFound com/google/...
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message