crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Deepak Subhramanian (JIRA)" <>
Subject [jira] [Created] (CRUNCH-220) Crunch
Date Mon, 17 Jun 2013 10:50:20 GMT
Deepak Subhramanian created CRUNCH-220:

             Summary: Crunch
                 Key: CRUNCH-220
             Project: Crunch
          Issue Type: Bug
          Components: IO
    Affects Versions: 0.6.0
         Environment: Cloudera Hadoop with Amazon S3
            Reporter: Deepak Subhramanian
            Priority: Minor

I am trying to use crunch to read file from S3 and write to S3. I am able to read the file
.But giving an error while writing to s3.  Not sure if it is a bug or I am missing a hadoop
configuration.  I am able to read from s3 and write to a local file or hdfs directly.  Here
is the code and error. I am passing s3 key and secret as parameters.  

PCollection<String> lines,   Writables.strings()));
    PCollection<String> textline = lines.parallelDo(new DoFn<String, String>()
        public void process(String line, Emitter<String> emitter) {
            if (headerNotWritten) {
                //emitter.emit("Writing Header");
                headerNotWritten =false;
            }else {
      }, Writables.strings()); // Indicates the serialization format

    pipeline.writeTextFile(textline, outputdir);

 Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: s3n://bktname/testcsv,
expected: hdfs://ip-address.compute.internal
[] out: 	at org.apache.hadoop.fs.FileSystem.checkPath(
[] out: 	at org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(
[] out: 	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(
[] out: 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(
[] out: 	at org.apache.hadoop.fs.FileSystem.exists(
[] out: 	at
[] out: 	at
[] out: 	at
[] out: 	at
[] out: 	at

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message