hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject Re: Dose hdfs support the configuration that different blocks can have different number of replcias?
Date Wed, 04 Mar 2015 21:50:53 GMT
Lipeng,

-setrep allows you to change the replication of an existing file. You can
also specify the replication factor when you initially create a file. Not
sure what you mean by "dynamically", that to me means calling setrep.

There is replication or invalidation work done as part of running -setrep.
This is done as a low-priority operation, unless the file is already in a
bad replication state (e.g. under-replicated).

Best,
Andrew

On Wed, Mar 4, 2015 at 12:18 PM, Lipeng Wan <lipengwan86@gmail.com> wrote:

> Hi Andrew,
>
> By using the -setrep command, can we change the replication factor of
> existing files? Or, can we change the replication factor of files
> dynamically? If that's possible, how much data movement overhead will
> occur?
> Thanks!
>
> Lipeng
>
> On Tue, Mar 3, 2015 at 2:57 PM, Andrew Wang <andrew.wang@cloudera.com>
> wrote:
> > Yup, definitely. Check out the -setrep command:
> >
> >
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep
> >
> > HTH,
> > Andrew
> >
> > On Tue, Mar 3, 2015 at 11:49 AM, Lipeng Wan <lipengwan86@gmail.com>
> wrote:
> >
> >> Hi Andrew,
> >>
> >> Thanks for your reply!
> >> Then is it possible for us to specify different replication factors
> >> for different files?
> >>
> >> Lipeng
> >>
> >> On Tue, Mar 3, 2015 at 2:38 PM, Andrew Wang <andrew.wang@cloudera.com>
> >> wrote:
> >> > Hi Lipeng,
> >> >
> >> > Right now that is unsupported, replication is set on a per-file basis,
> >> not
> >> > per-block.
> >> >
> >> > Andrew
> >> >
> >> > On Tue, Mar 3, 2015 at 11:23 AM, Lipeng Wan <lipengwan86@gmail.com>
> >> wrote:
> >> >
> >> >> Hi devs,
> >> >>
> >> >> By default, hdfs creates same number of replicas for each block. Is
> it
> >> >> possible for us to create more replicas for some of the blocks?
> >> >> Thanks!
> >> >>
> >> >> L. W.
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message