spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Don Drake <>
Subject Re: copy/mv hdfs file to another directory by spark program
Date Tue, 05 Jan 2016 03:52:03 GMT
You will need to use the HDFS API to do that.

Try something like:

val conf = sc.hadoopConfiguration
val fs = org.apache.hadoop.fs.FileSystem.get(conf)
fs.rename(new org.apache.hadoop.fs.Path("/path/on/hdfs/file.txt"), new

Full API for FileSystem is here:


On Mon, Jan 4, 2016 at 9:07 PM, Zhiliang Zhu <>

> For some file on hdfs, it is necessary to copy/move it to some another
> specific hdfs  directory, and the directory name would keep unchanged.
> Just need finish it in spark program, but not hdfs commands.
> Is there any codes, it seems not to be done by searching spark doc ...
> Thanks in advance!

Donald Drake
Drake Consulting <>

View raw message