hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Earl <charlesce...@me.com>
Subject Using virtual disk as HDFS storage
Date Sat, 10 Sep 2011 13:41:17 GMT
Hi,
I am exploring ways in which HDFS could be run in an environment supporting virtualization.
For the particular application, I would at least like the map tasks to run in virtual containers
(perhaps even lightweight containers for example LXC), but actually not duplicate the storage
per container. That is, one or more map task in the same container, as opposed to virtualized
datanode.
One option seems to be to have hadoop.tmp.dir as a virtual disk, use guestfs or similar for
management,  and respective map tasks could share this disk.
I'm eliding many details but just wanted some initial feedback, or pointers to similar effort.
I'm aware that Mesos (perhaps Hadoop NG?) has support for lxc.
Charles

Mime
View raw message