hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Youssef Hatem <yousef.ha...@gmail.com>
Subject Re: Getting a list of all StorageInfo
Date Thu, 08 Jan 2015 08:13:16 GMT
Hi Chris,

thanks a lot for taking the time answering my question. Skimming through
BlockPlacementPolicyDefault helped my a lot; I managed to get
DatanodeDescriptor(s) by using Host2NodesMap object. The DatanodeDescriptor
contains storage info which I was looking for.

Thanks again for your help, I really appreciate it.

Best regards,
Youssef

On Wed, Jan 7, 2015 at 10:21 PM, Chris Nauroth <cnauroth@hortonworks.com>
wrote:

> Hi Youssef,
>
> You might want to review the code for BlockPlacementPolicyDefault to get a
> sense for how an implementation works.  Perhaps you could even start your
> own work by copying and modifying that class.  This code would show you how
> to get ahold of a DatanodeDescriptor and then use it to call
> getStorageInfos.
>
> Hope this helps.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
> On Fri, Jan 2, 2015 at 10:12 AM, Youssef Hatem <yousef.hatem@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am working on implementing a custom placement policy. So I created a
> > class that extends BlockPlacementPolicy. I am trying to get a list of all
> > DatanodeStorageInfo objects in order to be able to decide which one to
> > choose for placement (based on certain criteria), this is the logic of
> > chooseTarget method.
> >
> > Could you please tell me how can I get this list? I tried to do the
> > following but I got none of these objects:
> >
> > DistributedFileSystem fs = (DistributedFileSystem)FileSystem.get(conf);
> > //conf is passed in initialize method
> > List<DatanodeStorageInfo> storageInfos = new ArrayList<>();
> > for(DatanodeInfo node :
> > fs.getDataNodeStats(HdfsConstants.DatanodeReportType.LIVE)){
> >   if(node instanceof DatanodeDescriptor){
> >
> >
> >
> storageInfos.addAll(Arrays.asList(((DatanodeDescriptor)node).getStorageInfo()));
> >   }
> >
> > It seems though that none of the resulting DatanodeInfo objects are
> > actually a DatanodeDescriptor therefor the storageInfos list is always
> > empty.
> >
> > It would be great if you could provide me with more information on how to
> > get a list of DatanodeStorageInfo objects.
> >
> > Thanks a lot in advance,
> > Youssef
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message