hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars Francke <lars.fran...@gmail.com>
Subject BlockPlacementPolicy question with hierarchical topology
Date Thu, 04 Jul 2019 09:15:11 GMT
Hi,

I have a customer who wants to make sure that copies of his data are
distributed amongst datacenters. So they are using rack names like this
/dc1/rack1, /dc1/rack2, /dc2/rack1 etc.

Unfortunately, the BlockPlacementPolicyDefault seems to place all blocks on
/dc1/* sometimes.

Is there a way to guarantee that /dc1/* and /dc2/* will be used in this
scenario?

Looking at chooseRandomWithStorageTypeTwoTrial it seems to consider the
full "scope" and not its components. I couldn't find anything in the code
but I had hoped I'm missing something: Is there a way to configure HDFS for
the behaviour I'd like?

Thanks!

Lars

Mime
View raw message