hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Filippo Spiga" <spiga.fili...@gmail.com>
Subject Integrate HADOOP and Map/Reduce paradigm into HPC environment
Date Sun, 17 Aug 2008 17:56:50 GMT
   I read a lot of tutorial and FAQ founded using search enginers but I
didn't find anything abount integration of HADOOP and Map/Reduce paradigm
into HPC environment. I manage a HPC cluster of 20 nodes (
http://scilx.disco.unimib.it/presentation-cluster-scilx ), used by 3
research group for their researches.

I want to integrate HADOOP into the cluster. My goals are:
- use all 20 nodes for HDFS filesystem
- select a subset of nodes to run Map/Reduce jobs
- submit Map/Reduce computation using TORQUE resource manager

I read the tutorial about HOD (Hadoop on demand) but HOD use torque only for
initial node allocation. I would use TORQUE also for "computation", allowing
users to load data into HDFS, submit a TORQUE JOB that execute a Map/Reduce
task and after retrive results. It's important for me that Map/Reduce tasks
run only on the subset of nodes selected by TORQUE.

Can someone help me?

Thanks in advance

Filippo Spiga
DISCo - FISLAB - Computational Physics and Complex Systems Laboratory
Rappresentante degli Studenti presso la Facoltà di Scienze Matematiche,
Fisiche e Naturali
Università degli Studi di Milano-Bicocca
mobile: +393408387735
Skype: filippo.spiga

C'e' un solo modo di dimenticare il tempo: impiegarlo.
-- Baudelaire, "Diari intimi"

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message