hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wellington Chevreuil <wellington.chevre...@gmail.com>
Subject Re: Create HDFS directory fails
Date Tue, 29 Jul 2014 08:57:07 GMT
Hum, I'm not sure, but I think through the API, you have to create each folder level at a time.
For instance, if your current path is "/user/logger" and you want to create "/user/logger/dev2/tmp2",
you have to first do hdfs.create(new Path("/user/logger/dev2")), then hdfs.create(new Path("/user/logger/dev2/tmp2")).
Have you already tried that?

On 29 Jul 2014, at 08:43, R J <rj201244@yahoo.com> wrote:

> Hi All,
> 
> I am trying to programmatically create a directory in HDFS but it fails with error.
> 
> This the part of my code:
> Path hdfsFile = new Path("/user/logger/dev2/tmp2");
> try {
> FSDataOutputStream out = hdfs.create(hdfsFile);
> }
> And I get this error:
> java.io.IOException: Mkdirs failed to create /user/logger/dev2/tmp2
>         at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:379)
>         at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:365)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:584)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:565)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:472)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:464)
>         at PutMerge.main(PutMerge.java:20)
> 
> I can create the same HDFS directory (and then remove) via hadoop command as the same
user who is running the java executable:
> $hadoop fs -mkdir /user/logger/dev/tmp2
> $hadoop fs -rmr /user/logger/dev/tmp2
> (above works)
> 
> Here is my entire code:
> ------PutMerge.java------
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> public class PutMerge {
> 
> public static void main(String[] args) throws IOException {
> Configuration conf = new Configuration();
> FileSystem hdfs = FileSystem.get(conf);
> FileSystem local = FileSystem.getLocal(conf);
> 
> Path inputDir = new Path("/home/tmp/test");
> Path hdfsFile = new Path("/user/logger/dev/tmp2");
> 
> try {
> FileStatus[] inputFiles = local.listStatus(inputDir);
> FSDataOutputStream out = hdfs.create(hdfsFile);
> 
> for (int i=0; i<inputFiles.length; i++) {
> System.out.println(inputFiles[i].getPath().getName());
> FSDataInputStream in = local.open(inputFiles[i].getPath());
> byte buffer[] = new byte[256];
> int bytesRead = 0;
> while( (bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
> in.close();
> }
> out.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> ------
> 


Mime
View raw message