hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Yang <lin.yang.ja...@gmail.com>
Subject Re: IOException when using MultipleSequenceFileOutputFormat
Date Mon, 17 Sep 2012 17:00:52 GMT
hey, Hien~

The datanode is running.

yanglin@ubuntu:~$ jps
> 14475 SecondaryNameNode
> 2642
> 14838 TaskTracker
> 14550 JobTracker
> 13877 NameNode
> 14177 DataNode
> 18811 Jps


I have found that if I use the MultipleSequenceFileOutputFormat instead
of LocalClusterMSFOutputFormat, this program works fine. here is my
inherited class which return a string according to the clustering result:

public class LocalClusterMSFOutputFormat extends
> MultipleSequenceFileOutputFormat<Text, RRIntervalWritable>
> {
> protected String generateFileNameForKeyValue(Text key,
>  RRIntervalWritable value, String name)
> {
>
> return value.clusterResult.toString();
>  }
> }


Is there any limitation of the path length or number of files in a single
directory?

2012/9/18 Hien Luu <hluu@yahoo.com>

> I ran into a similar problem the other day.  It turns out the datanode was
> not running.
>
> Type 'jps' to see if the datanode process is up and running.
>
> Hien
>
>   ------------------------------
> *From:* Jason Yang <lin.yang.jason@gmail.com>
> *To:* user@hadoop.apache.org
> *Sent:* Monday, September 17, 2012 6:50 AM
> *Subject:* IOException when using MultipleSequenceFileOutputFormat
>
> Hi, all
>
> I have written a simple MR program which partition a file into multiple
> files bases on the clustering result of the points in this file, here is my
> code:
> ---
> private int run() throws IOException
>  {
> String scheme = getConf().get(CommonUtility.ATTR_SCHEME);
> String ecgDir = getConf().get(CommonUtility.ATTR_ECG_DATA_DIR);
>  String outputDir = getConf().get(CommonUtility.ATTR_OUTPUT_DIR);
>
> // create JobConf
>  JobConf jobConf = new JobConf(getConf(), this.getClass());
>
> // set path for input and output
>  Path inPath = new Path(scheme + ecgDir);
> Path outPath = new Path(scheme + outputDir +
> CommonUtility.OUTPUT_LOCAL_CLUSTERING);
>  FileInputFormat.addInputPath(jobConf, inPath);
> FileOutputFormat.setOutputPath(jobConf, outPath);
>
> // clear output if it already existed
> CommonUtility.deleteHDFSFile(outPath.toString());
>
> // set format for input and output
> jobConf.setInputFormat(WholeFileInputFormat.class);
>  jobConf.setOutputFormat(LocalClusterMSFOutputFormat.class);
>
> // set class of output key and value
>  jobConf.setOutputKeyClass(Text.class);
> jobConf.setOutputValueClass(RRIntervalWritable.class);
>
> // set mapper and reducer
> jobConf.setMapperClass(LocalClusteringMapper.class);
>  jobConf.setReducerClass(IdentityReducer.class);
>
>
> // run the job
>  JobClient.runJob(jobConf);
> return 0;
> }
>
> ...
>
> public class LocalClusteringMapper extends MapReduceBase implements
> Mapper<NullWritable, BytesWritable, Text, RRIntervalWritable>
> {
> @Override
> public void map(NullWritable key, BytesWritable value,
>  OutputCollector<Text, RRIntervalWritable> output, Reporter reporter)
> throws IOException
>  {
> //read and cluster
>                   ...
>
> // output
>  Iterator<RRIntervalWritable> it = rrArray.iterator();
> while (it.hasNext())
>  {
> RRIntervalWritable rr = it.next();
>
> Text outputKey = new Text(rr.clusterResult );
>
> output.collect(outputKey, rr);
> }
>
> }
>
> ...
>
> public class LocalClusterMSFOutputFormat extends
> MultipleSequenceFileOutputFormat<Text, RRIntervalWritable>
> {
>
> protected String generateFileNameForKeyValue(Text key,
> RRIntervalWritable value, String name)
>  {
>  return value.clusterResult.toString();
>  }
> }
> ---
>
> But this program always get a IO Exception when running in a
> pseudo-distributed cluster, and the log has been attached at the end of
> this post.
>
> There's something wired:
> 1. If I use the SequenceFileOutputFormat instead of
> MultipleSequenceFileOutputFormat, this program would works fine( at least
> there is no error in log).
> 2. The one which always cause the error is the EcgData002509_LCF_3
>
>
>
> 12/09/17 21:10:35 INFO mapred.MapTask: Starting flush of map output
> 12/09/17 21:10:35 INFO mapred.MapTask: Finished spill 0
> 12/09/17 21:10:35 INFO mapred.TaskRunner:
> Task:attempt_local_0001_m_000019_0 is done. And is in the process of
> commiting
> 12/09/17 21:10:35 INFO mapred.LocalJobRunner:
> 12/09/17 21:10:35 INFO mapred.TaskRunner: Task
> 'attempt_local_0001_m_000019_0' done.
> 12/09/17 21:10:35 INFO mapred.LocalJobRunner:
> 12/09/17 21:10:35 INFO mapred.Merger: Merging 20 sorted segments
> 12/09/17 21:10:35 INFO mapred.Merger: Merging 2 intermediate segments out
> of a total of 20
> 12/09/17 21:10:35 INFO mapred.Merger: Merging 10 intermediate segments out
> of a total of 19
> 12/09/17 21:10:35 INFO mapred.Merger: Down to the last merge-pass, with 10
> segments left of total size: 18913891 bytes
> 12/09/17 21:10:35 INFO mapred.LocalJobRunner:
> 12/09/17 21:10:39 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /work/lab/output/localClustering/_temporary/_attempt_local_0001_r_000000_0/EcgData002509_LCF_3
> could only be replicated to 0 nodes, instead of 1
>  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> at org.apache.hadoop.ipc.Client.call(Client.java:740)
>  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
>  at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>  at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy0.addBlock(Unknown Source)
>  at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2937)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2819)
>  at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
> 12/09/17 21:10:39 WARN hdfs.DFSClient: Error Recovery for block null bad
> datanode[0] nodes == null
> 12/09/17 21:10:39 WARN hdfs.DFSClient: Could not get block locations.
> Source file
> "/work/lab/output/localClustering/_temporary/_attempt_local_0001_r_000000_0/EcgData002509_LCF_3"
> - Aborting...
> 12/09/17 21:10:41 INFO mapred.LocalJobRunner: reduce > reduce
> 12/09/17 21:10:42 INFO mapred.JobClient:  map 100% reduce 89%
> 12/09/17 21:10:42 WARN mapred.LocalJobRunner: job_local_0001
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /work/lab/output/localClustering/_temporary/_attempt_local_0001_r_000000_0/EcgData002509_LCF_3
> could only be replicated to 0 nodes, instead of 1
>  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> at org.apache.hadoop.ipc.Client.call(Client.java:740)
>  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
>  at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>  at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy0.addBlock(Unknown Source)
>  at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2937)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2819)
>  at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
> 12/09/17 21:10:43 INFO mapred.JobClient: Job complete: job_local_0001
> 12/09/17 21:10:43 INFO mapred.JobClient: Counters: 15
> 12/09/17 21:10:43 INFO mapred.JobClient:   FileSystemCounters
> 12/09/17 21:10:43 INFO mapred.JobClient:     FILE_BYTES_READ=23297226
> 12/09/17 21:10:43 INFO mapred.JobClient:     HDFS_BYTES_READ=546711709
> 12/09/17 21:10:43 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=232075142
> 12/09/17 21:10:43 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=13530993
> 12/09/17 21:10:43 INFO mapred.JobClient:   Map-Reduce Framework
> 12/09/17 21:10:43 INFO mapred.JobClient:     Reduce input groups=56
> 12/09/17 21:10:43 INFO mapred.JobClient:     Combine output records=0
> 12/09/17 21:10:43 INFO mapred.JobClient:     Map input records=20
> 12/09/17 21:10:43 INFO mapred.JobClient:     Reduce shuffle bytes=0
> 12/09/17 21:10:43 INFO mapred.JobClient:     Reduce output records=38837
> 12/09/17 21:10:43 INFO mapred.JobClient:     Spilled Records=102562
> 12/09/17 21:10:43 INFO mapred.JobClient:     Map output bytes=18691072
> 12/09/17 21:10:43 INFO mapred.JobClient:     Map input bytes=28649088
> 12/09/17 21:10:43 INFO mapred.JobClient:     Combine input records=0
> 12/09/17 21:10:43 INFO mapred.JobClient:     Map output records=55700
> 12/09/17 21:10:43 INFO mapred.JobClient:     Reduce input records=38838
> 12/09/17 21:10:44 INFO mapred.LocalJobRunner: reduce > reduce
> Exception in thread "main" java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>  at org.yanglin.mr.lab.ecg.PESCJob.runLocalClustering(PESCJob.java:111)
> at org.yanglin.mr.lab.ecg.PESCJob.run(PESCJob.java:57)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.yanglin.mr.lab.ecg.PESCJob.main(PESCJob.java:117)
> 12/09/17 21:10:48 ERROR hdfs.DFSClient: Exception closing file
> /work/lab/output/localClustering/_temporary/_attempt_local_0001_r_000000_0/EcgData002509_LCF_3
> : org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /work/lab/output/localClustering/_temporary/_attempt_local_0001_r_000000_0/EcgData002509_LCF_3
> could only be replicated to 0 nodes, instead of 1
>  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /work/lab/output/localClustering/_temporary/_attempt_local_0001_r_000000_0/EcgData002509_LCF_3
> could only be replicated to 0 nodes, instead of 1
>  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> at org.apache.hadoop.ipc.Client.call(Client.java:740)
>  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
>  at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>  at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy0.addBlock(Unknown Source)
>  at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2937)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2819)
>  at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
>
>
>
> --
> YANG, Lin
>
>
>
>


-- 
YANG, Lin

Mime
View raw message