hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <an...@orbita1.ru>
Subject RE: canot open *.crc file
Date Mon, 26 Jun 2006 08:04:12 GMT
My dfs don't create crc file...
Source code of my test class:
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.conf.Configuration;
import org.apache.nutch.util.NutchJob;
import org.apache.nutch.util.NutchConfiguration;

import java.io.IOException;

/**
 * @author Anton Potekhin
 * @date: 26.06.2006 11:48:24
 */

public class CreateFile {

    public static void main(String argv[]) {
        if ("create".equals(argv[0])) {
            Configuration conf = NutchConfiguration.create();
            JobConf job = new NutchJob(conf);
            try {
                FileSystem fs = FileSystem.get(job);
                fs.createNewFile(new Path("test", "done"));
            } catch (IOException e) {
                System.err.println(e.toString());
                return;
            }
        }
        if ("get".equals(argv[0])) {
            Configuration conf = NutchConfiguration.create();
            JobConf job = new NutchJob(conf);
            try {
                FileSystem fs = FileSystem.get(job);
                fs.copyToLocalFile(new Path("test", "done"),new
Path("done"));
            } catch (IOException e) {
                System.err.println(e.toString());
                return;
            }
        }

    }
}


1) Start this class with parameter "create".
2) Then:
	# bin/hadoop dfs -ls test
	Found 1 items
	/user/root/test/done    <r 2>   0

	And I don't see crc file...
3) Start test class with parameter "get". In results I get file done and
.done.crc in local filesystem and error:

060626 010145 Client connection to 127.0.0.1:9000: starting
060626 010145 Problem opening checksum file: test/done.  Ignoring with
exception org.apache.hadoop.ipc.RemoteException: java.io.IOException: Cannot
open filename /user/root/test/.done.crc
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:130)
        at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:240)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:218)
.
	

Why I see this error? 


-----Original Message-----
From: Konstantin Shvachko [mailto:shv@yahoo-inc.com] 
Sent: Saturday, June 24, 2006 2:41 AM
To: hadoop-user@lucene.apache.org
Subject: Re: canot open *.crc file
Importance: High

The crc files are generated automatically, you do not need to create them.
It looks like you do everything right, and it should work.
Are you trying to read the crc files?
The crc files name pattern is ".fname.crc"
in you case it should be ".done.crc" rather than "done.crc"
--Konstantin

anton@orbita1.ru wrote:

>I create file on dfs (for example filename "done"). After I try copy this
>file from dfs to local filesystem. In result I get this file in local
>filesystem and error:
>
>Problem opening checksum file: /user/root/crawl/done.  Ignoring with
>exception org.apache.hadoop.ipc.RemoteException: jav
>a.io.IOException: Cannot open filename /user/root/crawl/done.crc
>        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:130)
>        at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
>        at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:240)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:218) 
>
>
>for create file I use code:
>	FileSystem fs = ...
>	fs_.createNewFile(new Path(segments[i], "already_indexed"));
>
>for copy file to local filesystem I use code:
>	fs.copyToLocalFile(...,...);
>
>How create crc file? 
>Why crc file is not created automatically when making a file on dfs? 
>How correctly create a file on dfs?
>  
>




Mime
View raw message