Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B90654328 for ; Tue, 21 Jun 2011 08:44:36 +0000 (UTC) Received: (qmail 59157 invoked by uid 500); 21 Jun 2011 08:44:33 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 59046 invoked by uid 500); 21 Jun 2011 08:44:32 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 59038 invoked by uid 99); 21 Jun 2011 08:44:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 08:44:32 +0000 X-ASF-Spam-Status: No, hits=3.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of avivaknin13@gmail.com designates 74.125.82.176 as permitted sender) Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 08:44:23 +0000 Received: by wyb36 with SMTP id 36so1801388wyb.35 for ; Tue, 21 Jun 2011 01:44:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:subject:date:message-id:mime-version :content-type:x-mailer:thread-index:content-language; bh=oyziF5M8IVd/7qpLw5/gNBxI9zj4EqJSo3wHxhXqcZo=; b=UyAPfREdCySFJ/7JTPfwOSfNneoTowN6omfTRL8YmQ/0u3rCO5Sl8oZkz83/M/ptwV DhWL+z2kqDkLchsLEHwVPhCAbJt6qKYa3FOKrdC7S9XE3/9E9Ubqf6N4wLLnulIxQEpB nOCsV8qqnCRXqc5sdWo0CrQJzrywgLhFb/BLQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:subject:date:message-id:mime-version:content-type:x-mailer :thread-index:content-language; b=wC9ySMmTRciMQzVpiJTQVhTfvap97i+1yyRavCo1xabEPX+fuMjXKWYquFVfAuBqSz OY4HlDcGKknjT2v6GRTNmdEk+gqWiMTqUdXYrIzOS5lYzFrT8PZcyhHUk5W0ssOdyWg0 yKmpHekcP2pHDcRti80tt0Rtz+3Vy1HRWDcGU= Received: by 10.216.120.129 with SMTP id p1mr6233255weh.81.1308645843151; Tue, 21 Jun 2011 01:44:03 -0700 (PDT) Received: from AviPC ([2.54.57.18]) by mx.google.com with ESMTPS id b70sm3335636wek.44.2011.06.21.01.44.00 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 21 Jun 2011 01:44:01 -0700 (PDT) From: "Avi Vaknin" To: Subject: Help with adjusting Hadoop configuration files Date: Tue, 21 Jun 2011 11:43:53 +0300 Message-ID: <039301cc2fef$5d2496f0$176dc4d0$@com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0394_01CC3008.8271CEF0" X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acwv71mH3on/GmG2SXSwpc2LxW01xA== Content-Language: he X-Virus-Checked: Checked by ClamAV on apache.org ------=_NextPart_000_0394_01CC3008.8271CEF0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi Everyone, We are a start-up company has been using the Hadoop Cluster platform (version 0.20.2) on Amazon EC2 environment. We tried to setup a cluster using two different forms: Cluster 1: includes 1 master (namenode) + 5 datanodes - all of the machines are small EC2 instances (1.6 GB RAM) Cluster 2: includes 1 master (namenode) + 2 datanodes - the master is a small EC2 instance and the other two datanodes are large EC2 instances (7.5 GB RAM) We tried to make changes on the the configuration files (core-sit, hdfs-site and mapred-sit xml files) and we expected to see a significant improvement on the performance of the cluster 2, unfortunately this has yet to happen. Are there any special parameters on the configuration files that we need to change in order to adjust the Hadoop to a large hardware environment ? Are there any best practice you recommend? Thanks in advance. Avi ------=_NextPart_000_0394_01CC3008.8271CEF0--