hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Varma <susheel.va...@gmail.com>
Subject Metadata, Daemons, Benchmarks, Web UI
Date Thu, 25 Feb 2010 06:39:56 GMT

We are trying evaluate a small set of distributed data management
solutions(iRODS, HDFS, Lustre) for our project. We don't really have a
need for scalable computation, but rather our focus is more on
redundancy, reliability and security. Although small bits computation
would be needed at some level. We have just begun an evaluation of
HDFS, however this has thrown up a few questions(a good thing, I

1. Metadata & Links
a. Is there a way to add/update/remove metadata held on the NameNode? Examples?
b. Is there a way to get hold of the metadata held on the NameNode?
I'd like to allow users to search the HDFS using the the metadata, and
then resolve the query to actual link to the file.

If a or b is not possible, I would have to resort to using an HBase to
store the custom metadata. Examples?

2. Daemons
a. Is there a way to setup a daemon job(map-reduce, or otherwise) to
listen for filesystem events and trigger actions that need to be
performed on these files? Examples?
b. If not, Is there an FSEvent API I could use? Examples?

3. Benchmarks
a. Are there any un/published HDFS filesystem benchmarks using IOZone,
PostMark etc. I know almost all FS benchmarks must be taken with a
pinch of salt, but I'd really like to see the quantitative comparisons
with Lustre for example.

4. Web UI
a. Is there a simple way to augment the NameNode Web UI to allow users
to login, search and download the files on the backend filesystem?
b. If not, could you show me examples where users have combined a web
application to serve files store on the HDFS?


View raw message