hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravi teja <raviort...@gmail.com>
Subject Bulk chmod,chown operations on HDFS
Date Thu, 16 Jun 2016 03:33:35 GMT
Hi Community,

As part of the new authorisation changes, we need to change the permissions
and owners of many files in hdfs (2.6.0) with chmod and chown.

To do this we need to stop the processing on the directories to avoid
inconsistencies in permissions, hence we need to take a downtime for those
specific pipelines operating on these folders.


The total number of files/directories to be operated upon is around 10
Million.
A chmod recursive (chmod -R) on 160K objects, has taken around 15 minutes.

At this rate it will take a long time to complete the operation and the
downtime would be couple of hours.

Mapreduce program  is one option, but chmod,chown being a heavy operations,
will slow down the cluster for other users, if done at this scale.

Are there any options to do a bulk permissions changes chmod,chown to avoid
these issues?
If not are there any alternative approaches to carry the same operation at
this scale something like admin backdoor to fsimage?



Thanks,
Ravi Teja

Mime
View raw message