Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 19847 invoked from network); 10 May 2009 21:30:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 10 May 2009 21:30:44 -0000 Received: (qmail 35776 invoked by uid 500); 10 May 2009 21:30:42 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 35676 invoked by uid 500); 10 May 2009 21:30:41 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 35666 invoked by uid 99); 10 May 2009 21:30:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 May 2009 21:30:41 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of mattbowyers@googlemail.com designates 209.85.200.174 as permitted sender) Received: from [209.85.200.174] (HELO wf-out-1314.google.com) (209.85.200.174) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 May 2009 21:30:30 +0000 Received: by wf-out-1314.google.com with SMTP id 23so1910451wfg.2 for ; Sun, 10 May 2009 14:30:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=P8Ol4K2YIgIBSbwOH+ShtEplBa99aEQPRfKTZhOSq8I=; b=Rnpew3adMf97beXy0FvVUBdiljaIAvT3w8zG3qurdfSrFByfDTv85vSN1FXjxS7NmB J3+z86or3BTB1w0h7cFHqJQMJs9jrOCYkRXU4995THqZ6eecsMtu/rvXzqhd19/kPaC7 BxKq0GMaklClaCbzIE5KNIPwWFOL1DkKFpLwY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=ivZ50eLWPaKS38BENlYTl5pt4IN67rg6jG0e7HRetZYqxgC0qBqKPdE05TqDvuEBFG g2NGVkYx3/0yKdYqnJVImACPjUiTJaqLqUuAw1gioUbTujIOEZdiMJRaxAIzMALYPYer JHJikYCk95XmVODNZDrvR1CwZiPs+6TOFHUik= MIME-Version: 1.0 Received: by 10.142.229.6 with SMTP id b6mr2740445wfh.98.1241991010378; Sun, 10 May 2009 14:30:10 -0700 (PDT) Date: Sun, 10 May 2009 22:30:10 +0100 Message-ID: Subject: sub 60 second performance From: Matt Bowyer To: core-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=000e0cd32c48fce45704699590f9 X-Virus-Checked: Checked by ClamAV on apache.org --000e0cd32c48fce45704699590f9 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi, I am trying to do 'on demand map reduce' - something which will return in reasonable time (a few seconds). My dataset is relatively small and can fit into my datanode's memory. Is it possible to keep a block in the datanode's memory so on the next job the response will be much quicker? The majority of the time spent during the job run appears to be during the 'HDFS_BYTES_READ' part of the job. I have tried using the setNumTasksToExecutePerJvm but the block still seems to be cleared from memory after the job. thanks! --000e0cd32c48fce45704699590f9--