hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-535) TestUnmanagedAMLauncher can corrupt target/test-classes/yarn-site.xml during write phase, breaks later test runs
Date Wed, 03 Apr 2013 14:59:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620974#comment-13620974
] 

Steve Loughran commented on YARN-535:
-------------------------------------

{code}
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher  Time elapsed:
4137 sec  <<< ERROR!
java.lang.RuntimeException: Error parsing 'yarn-site.xml' : org.xml.sax.SAXParseException:
Premature end of file.
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2050)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1899)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1816)
	at org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:465)
	at org.apache.hadoop.conf.Configuration.asXmlDocument(Configuration.java:2127)
	at org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:2096)
	at org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:2086)
	at org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.setup(TestUnmanagedAMLauncher.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:246)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
	at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:153)
	at org.apache.hadoop.conf.Configuration.parse(Configuration.java:1887)
	at org.apache.hadoop.conf.Configuration.parse(Configuration.java:1875)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1946)
	... 29 more
{code}
This stack trace is a failure to read the file yarn-site.xml, which is actually being written
on line 63 of TestUnmanagedAMLauncher -a file that 
is already open for writing. 

It is possible that some filesystems (here, HFS+) make that write visible while it is still
going on, triggering a failure which then corrupts later builds at init time

{code}
$ ls -l target/test-classes/yarn-site.xml 
-rw-r--r--  1 stevel  staff  0  3 Apr 15:37 target/test-classes/yarn-site.xml
{code}

This is newer than the one in test/properties, so Maven doesn't fix it next test run
{code}
$ ls -l src/test/resources/yarn-site.xml 
-rw-r--r--@ 1 stevel  staff  830 28 Nov 16:29 src/test/resources/yarn-site.xml
{code}
as a result, follow on tests fail when MiniYARNCluster tries to read it.

{code}
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher  Time elapsed:
515 sec  <<< ERROR!
java.lang.RuntimeException: Error parsing 'yarn-site.xml' : org.xml.sax.SAXParseException:
Premature end of file.
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2050)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1899)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1816)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:719)
	at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:738)
	at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:948)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:62)
	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:244)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:213)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:674)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:582)
	at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:458)
	at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:439)
	at org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:425)
	at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:93)
	at org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.setup(TestUnmanagedAMLauncher.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
{code}

# This test should copy over the new yarn-site.xml rather than overwrite the live one
# there should be a yarn-site-template.xml that is copied over
# the {{Configuration.loadResource()}} could skip size 0 files instead of trying to parse
them.



                
> TestUnmanagedAMLauncher can corrupt target/test-classes/yarn-site.xml during write phase,
breaks later test runs
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-535
>                 URL: https://issues.apache.org/jira/browse/YARN-535
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: applications
>    Affects Versions: 3.0.0
>         Environment: OS/X laptop, HFS+ filesystem
>            Reporter: Steve Loughran
>            Priority: Minor
>
> the setup phase of {{TestUnmanagedAMLauncher}} overwrites {{yarn-site.xml}}. As {{Configuration.writeXml()}}
does a reread of all resources, this will break if the (open-for-writing) resource is already
visible as an empty file. 
> This leaves a corrupted {{target/test-classes/yarn-site.xml}}, which breaks later test
runs -because it is not overwritten by later incremental builds, due to timestamps.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message