beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Created] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file
Date Wed, 29 Nov 2017 23:16:00 GMT
Eugene Kirpichov created BEAM-3272:

             Summary: ParDoTranslatorTest: Error creating local cluster while creating checkpoint
                 Key: BEAM-3272
             Project: Beam
          Issue Type: Bug
          Components: runner-apex
    Affects Versions: 2.3.0
            Reporter: Eugene Kirpichov
            Assignee: Thomas Weise
            Priority: Minor

Failed build:$beam-runners-apex/5330/console

Key output:

2017-11-29T01:21:26.956 [ERROR] testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
 Time elapsed: 2.007 s  <<< ERROR!
java.lang.RuntimeException: Error creating local cluster
	at org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(
	at org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(
	at org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(
	at org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(
Caused by: ExitCodeException exitCode=1: chmod: cannot access ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
No such file or directory

	at org.apache.hadoop.util.Shell.runCommand(
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
	at org.apache.hadoop.util.Shell.execCommand(
	at org.apache.hadoop.util.Shell.execCommand(
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(
	at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(
	at org.apache.hadoop.fs.RawLocalFileSystem.create(
	at org.apache.hadoop.fs.RawLocalFileSystem.create(
	at org.apache.hadoop.fs.FileSystem.primitiveCreate(
	at org.apache.hadoop.fs.DelegateToFileSystem.createInternal(
	at org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.<init>(
	at org.apache.hadoop.fs.ChecksumFs.createInternal(
	at org.apache.hadoop.fs.AbstractFileSystem.create(
	at org.apache.hadoop.fs.FileContext$
	at org.apache.hadoop.fs.FileContext$
	at org.apache.hadoop.fs.FSLinkResolver.resolve(
	at org.apache.hadoop.fs.FileContext.create(
	at com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(
	... 50 more

By inspecting code at the stack frames, seems it's trying to copy an operator's checkpoint
"to HDFS" (which in this case is the local disk), but fails while creating the target file
of the copy - creation creates the file (successfully) and chmods it writable (unsuccessfully).
Barring something subtle (e.g. chmod being not allowed to call immediately after creating
a FileOutputStream), this looks like the whole directory was possibly deleted from under the
process. I don't know why this would be the case though, or how to debug it.

Either way, the path being accessed is funky: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/...
- I think it'd be better if this test used a "@Rule TemporaryFolder" to store Apex checkpoints.
I don't know whether the Apex runner allows that, but I can see how it could help reduce interference
between tests and potentially resolve this issue.

This message was sent by Atlassian JIRA

View raw message