hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Earl <charlesce...@me.com>
Subject Using virtual disk as HDFS storage
Date Sat, 10 Sep 2011 13:41:17 GMT
I am exploring ways in which HDFS could be run in an environment supporting virtualization.
For the particular application, I would at least like the map tasks to run in virtual containers
(perhaps even lightweight containers for example LXC), but actually not duplicate the storage
per container. That is, one or more map task in the same container, as opposed to virtualized
One option seems to be to have hadoop.tmp.dir as a virtual disk, use guestfs or similar for
management,  and respective map tasks could share this disk.
I'm eliding many details but just wanted some initial feedback, or pointers to similar effort.
I'm aware that Mesos (perhaps Hadoop NG?) has support for lxc.

View raw message