ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-3312) [Test] HadoopMapReduceEmbeddedSelfTest.testMultiReducerWholeMapReduceExecution flakily fails.
Date Tue, 28 Jun 2016 08:45:57 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vladimir Ozerov updated IGNITE-3312:
------------------------------------
    Component/s:     (was: IGFS)
                 hadoop

> [Test] HadoopMapReduceEmbeddedSelfTest.testMultiReducerWholeMapReduceExecution flakily
fails.
> ---------------------------------------------------------------------------------------------
>
>                 Key: IGNITE-3312
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3312
>             Project: Ignite
>          Issue Type: Bug
>          Components: hadoop
>            Reporter: Ivan Veselovsky
>            Assignee: Vladimir Ozerov
>              Labels: test
>
> HadoopMapReduceEmbeddedSelfTest.testMultiReducerWholeMapReduceExecution fails with ~20%
probability . 
> Failure cause is either the 1st or 2nd marked line in the following code (org.apache.ignite.internal.processors.igfs.IgfsMetaManager#create)
:
> {code}
>                             // Check: can we overwrite it?
>                             if (!overwrite)
>                                 throw new IgfsPathAlreadyExistsException("Failed to create
a file: " + path); // ***** #1
>                             // Check if file already opened for write.
>                             if (oldInfo.lockId() != null)
>                                 throw new IgfsException("File is already opened for write:
" + path); // ***** #2
> {code}
> Diagnostic shows that the same file really attempted to be created several times on one
thread.
> Situation #2 reproducible with the following stack when IGFS is used as a secondary fs:
> {code}
> Hadoop-task-efea2ae1-09cb-4c49-9465-dcbbedee1835_1-REDUCE-2-0-#679%hadoop.HadoopMapReduceEmbeddedSelfTest2%@20675,
prio=5, in group 'ignite', status: 'RUNNING'
>           at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.create(IgfsMetaManager.java:2857)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1051)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1019)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1823)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:1019)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:990)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:359)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:357)
>           at org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:49)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.create(HadoopIgfsInProc.java:357)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:258)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:255)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:310)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.create(HadoopIgfsWrapper.java:255)
>           at org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.create(IgniteHadoopFileSystem.java:632)
>           at org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.create(IgniteHadoopIgfsSecondaryFileSystem.java:406)
>           at org.apache.ignite.internal.processors.igfs.IgfsSecondaryFileSystemCreateContext.create(IgfsSecondaryFileSystemCreateContext.java:87)
>           at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.create(IgfsMetaManager.java:2922)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1051)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1019)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1823)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:1019)
>           at org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:990)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:359)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:357)
>           at org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:54)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.create(HadoopIgfsInProc.java:357)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:258)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:255)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:310)
>           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.create(HadoopIgfsWrapper.java:255)
>           at org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.create(IgniteHadoopFileSystem.java:632)
>           at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
>           at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:799)
>           at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
>           at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1OutputCollector.<init>(HadoopV1OutputCollector.java:68)
>           at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1Task$1.<init>(HadoopV1Task.java:73)
>           at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1Task.collector(HadoopV1Task.java:72)
>           at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1ReduceTask.run(HadoopV1ReduceTask.java:65)
>           at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:249)
>           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:201)
>           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0(HadoopRunnableTask.java:144)
>           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:116)
>           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:114)
>           at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.runAsJobOwner(HadoopV2TaskContext.java:544)
>           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:114)
>           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:46)
>           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:186)
>           at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>           at java.lang.Thread.run(Thread.java:745)
> {code}
> situation #1 also happens when IGFS plays role of the 2ndary file system with the following
stack :
> {code}
>   72 Hadoop-task-aebecef6-5cf2-4b66-9674-9fd8644e1183_1-REDUCE-0-0-#368%hadoop.HadoopMapReduceEmbeddedSelfTest0%@14531,
prio=5, in group 'ignite', status: 'RUNNING'
>   73           at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.create(IgfsMetaManager.java:2853)
>   74           at org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1051)
>   75           at org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1019)
>   76           at org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1823)
>   77           at org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:1019)
>   78           at org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:990)
>   79           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:359)
>   80           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:357)
>   81           at org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:49)
>   82           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.create(HadoopIgfsInProc.java:357)
>   83           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:258)
>   84           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:255)
>   85           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:310)
>   86           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.create(HadoopIgfsWrapper.java:255)
>   87           at org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.create(IgniteHadoopFileSystem.java:632)
>   88           at org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.create(IgniteHadoopIgfsSecondaryFileSystem.java:406)
>   89           at org.apache.ignite.internal.processors.igfs.IgfsSecondaryFileSystemCreateContext.create(IgfsSecondaryFileSystemCreateContext.java:87)
>   90           at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.create(IgfsMetaManager.java:2922)
>   91           at org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1051)
>   92           at org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1019)
>   93           at org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1823)
>   94           at org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:1019)
>   95           at org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:990)
>   96           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:359)
>   97           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:357)
>   98           at org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:54)
>   99           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.create(HadoopIgfsInProc.java:357)
>  100           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:258)
>  101           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:255)
>  102           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:310)
>  103           at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.create(HadoopIgfsWrapper.java:255)
>  104           at org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.create(IgniteHadoopFileSystem.java:632)
>  105           at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
>  106           at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
>  107           at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
>  108           at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:132)
>  109           at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.prepareWriter(HadoopV2Task.java:103)
>  110           at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2ReduceTask.run0(HadoopV2ReduceTask.java:57)
>  111           at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.run(HadoopV2Task.java:54)
>  112           at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:249)
>  113           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:201)
>  114           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0(HadoopRunnableTask.java:144)
>  115           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:116)
>  116           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:114)
>  117           at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.runAsJobOwner(HadoopV2TaskContext.java:544)
>  118           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:114)
>  119           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:46)
>  120           at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:186)
>  121           at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>  122           at java.lang.Thread.run(Thread.java:745)
> {code} 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message