ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Veselovsky (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-418) "terasort" example does not work on secondary HDFS with DUAL_SYNC, DUAL_ASYNC mode
Date Fri, 06 Mar 2015 18:57:42 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ivan Veselovsky updated IGNITE-418:
-----------------------------------
    Attachment: IGNITE-418--a.patch

> "terasort" example does not work on secondary HDFS with DUAL_SYNC, DUAL_ASYNC mode
> ----------------------------------------------------------------------------------
>
>                 Key: IGNITE-418
>                 URL: https://issues.apache.org/jira/browse/IGNITE-418
>             Project: Ignite
>          Issue Type: Bug
>          Components: hadoop
>    Affects Versions: sprint-1
>            Reporter: Ivan Veselovsky
>            Assignee: Ivan Veselovsky
>            Priority: Critical
>             Fix For: sprint-2
>
>         Attachments: IGNITE-418--a.patch
>
>
> On attempt to run "terasort" Hadoop example on top of IGFS backed by HDFS in DUAL_SYNC,
DUAL_ASYNC mode we have failure: hadoop job finished Ok, but its fesult files are not found:
> + ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar teragen
1000000 /tmp/teragen
> 15/03/05 19:07:26 INFO terasort.TeraSort: Generating 1000000 using 2
> 15/03/05 19:07:26 INFO mapreduce.JobSubmitter: number of splits:2
> 15/03/05 19:07:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
> 15/03/05 19:07:27 INFO mapreduce.Job: The url to track the job: N/A
> 15/03/05 19:07:27 INFO mapreduce.Job: Running job: job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
> 15/03/05 19:07:29 INFO mapreduce.Job: Job job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
running in uber mode : false
> 15/03/05 19:07:29 INFO mapreduce.Job:  map 0% reduce 0%
> 15/03/05 19:07:31 INFO mapreduce.Job:  map 50% reduce 0%
> 15/03/05 19:07:31 INFO mapreduce.Job:  map 100% reduce 100%
> 15/03/05 19:07:31 INFO mapreduce.Job: Job job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
completed successfully
> 15/03/05 19:07:31 INFO mapreduce.Job: Counters: 1
>         org.apache.hadoop.examples.terasort.TeraGen$Counters
>                 CHECKSUM=2148987642402270
> + ./hadoop fs -ls /tmp/teragen
> ls: `/tmp/teragen': No such file or directory
> At the same time the following exception is on the Ignite node: 
> [19:07:16] Topology snapshot [ver=1, nodes=1, CPUs=8, heap=1.0GB]
> [19:07:31,288][ERROR][Hadoop-task-774fb810-21a7-48d9-81d1-ce1ac727841f_1-COMMIT-0-0-#110%null%][IgfsMetaManager]
Path rename in DUAL mode failed [source=/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000,
destination=/tmp/teragen]
> class org.apache.ignite.IgniteCheckedException: Failed to rename (source path not found):
/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000
>         at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.fsException(IgfsMetaManager.java:2668)
>         at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.access$1100(IgfsMetaManager.java:53)
>         at org.apache.ignite.internal.processors.igfs.IgfsMetaManager$7.onSuccess(IgfsMetaManager.java:2052)
>         at org.apache.ignite.internal.processors.igfs.IgfsMetaManager$7.onSuccess(IgfsMetaManager.java:2043)
>         at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2527)
>         at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2337)
>         at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.renameDual(IgfsMetaManager.java:2103)
>         at org.apache.ignite.internal.processors.igfs.IgfsImpl.rename(IgfsImpl.java:673)
>         at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.rename(HadoopIgfsInProc.java:128)
>         at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$5.apply(HadoopIgfsWrapper.java:133)
>         at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$5.apply(HadoopIgfsWrapper.java:130)
>         at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:289)
>         at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.rename(HadoopIgfsWrapper.java:130)
>         at org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.rename(IgniteHadoopFileSystem.java:698)
>         at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:345)
>         at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
>         at org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
>         at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:56)
>         at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.igfs.IgfsFileNotFoundException: Failed to rename (source
path not found): /tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000
>         ... 23 more
> [19:07:31,294][ERROR][Hadoop-task-774fb810-21a7-48d9-81d1-ce1ac727841f_1-COMMIT-0-0-#110%null%][HadoopRunnableTask]
Task execution failed.
> class org.apache.ignite.IgniteCheckedException: class org.apache.ignite.IgniteCheckedException:
Failed to rename FileStatus [path=igfs://igfs@localhost/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000,
isDir=false, len=0, mtime=1425571651255, atime=1425571651255] to igfs://igfs@localhost/tmp/teragen
>         at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:59)
>         at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Failed to rename FileStatus [path=igfs://igfs@localhost/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000,
isDir=false, len=0, mtime=1425571651255, atime=1425571651255] to igfs://igfs@localhost/tmp/teragen
>         at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:346)
>         at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
>         at org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
>         at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:56)
>         ... 7 more
>         at org.apache.ignite.internal.processors.hadoop.HadoopUtils.transformException(HadoopUtils.java:273)
>         at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:196)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
>         at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message