Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 84936 invoked from network); 17 Aug 2008 17:57:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 17 Aug 2008 17:57:27 -0000 Received: (qmail 84857 invoked by uid 500); 17 Aug 2008 17:57:21 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 84820 invoked by uid 500); 17 Aug 2008 17:57:21 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 84809 invoked by uid 99); 17 Aug 2008 17:57:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Aug 2008 10:57:21 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of spiga.filippo@gmail.com designates 64.233.184.232 as permitted sender) Received: from [64.233.184.232] (HELO wr-out-0506.google.com) (64.233.184.232) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Aug 2008 17:56:22 +0000 Received: by wr-out-0506.google.com with SMTP id c30so1986865wra.21 for ; Sun, 17 Aug 2008 10:56:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=M4gbJSeJc7Q4dcL3kfMQqDTVYKFufzfwXCfxmZeKCg4=; b=t/88lEPLF4E2xE+He5CIBI/xtngaLr6bPoEO3m/k0L8KfVVPSdeHIJdW4CWjT8TXe0 ahWjl1e09EdzDuXHDyLHJQ4Cx2mK7C1kfuvxNpfesN3T4Mncd+9kCit8uSawjHXnG845 sPc2MVA/2F6eVYJJTKI/rSErtjYkH6H0Vd2aM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=kf/H6DFr21K/OaMVX6/crvx7nBRRBlZ4qIm3KjelFPZOkvQzaH7Te5C3HBgK2I/FDr WxqN9GheUS6Af+rFkbZ+q3Y7tMhEbRX/IbfqNP9AiQ3iPiGtjEodxejahOZvoIBEDrMs XCmPcJvAnBMZOjwVJXpNii+bT4V0pZxGwxWRA= Received: by 10.90.71.15 with SMTP id t15mr4529589aga.14.1218995810227; Sun, 17 Aug 2008 10:56:50 -0700 (PDT) Received: by 10.90.68.15 with HTTP; Sun, 17 Aug 2008 10:56:50 -0700 (PDT) Message-ID: Date: Sun, 17 Aug 2008 19:56:50 +0200 From: "Filippo Spiga" To: core-user@hadoop.apache.org Subject: Integrate HADOOP and Map/Reduce paradigm into HPC environment MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_19120_4751417.1218995810222" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_19120_4751417.1218995810222 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi, I read a lot of tutorial and FAQ founded using search enginers but I didn't find anything abount integration of HADOOP and Map/Reduce paradigm into HPC environment. I manage a HPC cluster of 20 nodes ( http://scilx.disco.unimib.it/presentation-cluster-scilx ), used by 3 research group for their researches. I want to integrate HADOOP into the cluster. My goals are: - use all 20 nodes for HDFS filesystem - select a subset of nodes to run Map/Reduce jobs - submit Map/Reduce computation using TORQUE resource manager I read the tutorial about HOD (Hadoop on demand) but HOD use torque only fo= r initial node allocation. I would use TORQUE also for "computation", allowin= g users to load data into HDFS, submit a TORQUE JOB that execute a Map/Reduce task and after retrive results. It's important for me that Map/Reduce tasks run only on the subset of nodes selected by TORQUE. Can someone help me? Thanks in advance --=20 Filippo Spiga DISCo - FISLAB - Computational Physics and Complex Systems Laboratory Rappresentante degli Studenti presso la Facolt=E0 di Scienze Matematiche, Fisiche e Naturali Universit=E0 degli Studi di Milano-Bicocca mobile: +393408387735 Skype: filippo.spiga C'e' un solo modo di dimenticare il tempo: impiegarlo. -- Baudelaire, "Diari intimi" ------=_Part_19120_4751417.1218995810222--