hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: possible to change replication factor at file creation time (with copyFromLocal)?
Date Fri, 31 May 2013 17:03:31 GMT
Hi Julian,

Yes, "dfs" subcommand accepts config overrides via -D. Just do "hadoop
dfs -Ddfs.replication=X -copyFromLocal …".

On Fri, May 31, 2013 at 10:27 PM, Julian Bui <julianbui@gmail.com> wrote:
> Hi hadoop users,
> I am aware that you can set the replication factor of a file after it's been
> created, but can you do it as you copy files to the HDFS?  My hope/intuition
> is that if you were able to reduce the replication factor of a file while
> copying, the copy time would decrease.  I'm finding it difficult waiting for
> large data sets to copy over.
> I am currently doing:
> hadoop dfs -copyFromLocal "/copy/from/path/" input
> and am wondering if it's possible to also specify something like -setrep on
> the same line.  -setsrep requires you to specify the file, which implies
> that it has to exist first, requiring two separate commands.
> Thanks in advance,
> -Julian

Harsh J

View raw message