Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2999110A9E for ; Mon, 24 Feb 2014 10:36:37 +0000 (UTC) Received: (qmail 71173 invoked by uid 500); 24 Feb 2014 10:36:28 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 71016 invoked by uid 500); 24 Feb 2014 10:36:25 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 71003 invoked by uid 99); 24 Feb 2014 10:36:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Feb 2014 10:36:24 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of drdwitte@gmail.com designates 209.85.215.51 as permitted sender) Received: from [209.85.215.51] (HELO mail-la0-f51.google.com) (209.85.215.51) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Feb 2014 10:36:19 +0000 Received: by mail-la0-f51.google.com with SMTP id c6so5420518lan.10 for ; Mon, 24 Feb 2014 02:35:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=FWfNa5eCdRYbEjirB7Fw6xqHmLGT8r/lcfWhFk0s3w4=; b=PtrpqjDM5tp/6hfn5SB1ZoKwObZ69r6X54izjzVjRE+w74OOQalIqFDml5fOkxHyYE TV7b1oLKdyNBWLNTp9oo9yS9ctiR4oTytKxH2LHSWihdx+aea9ldj2iajBWrfx0m/VIk jsTChfUM7+ms/5rLeTpvh7W/ZSh/6uG8qSwd8AvhPXoxlcqWG9P844aFfTHqN+Mf5nij /+iomUHpsxvArybfLbiNa2mdDBRLu0bjfJ+o9fE5bbnYtBWIIkvqN1v2Bg7+c+b13Wz+ A9y3I/7f6YZELd5lkwjf3SAO7aVlNi4SwYjeYSu4ZHIDh2F/S1b8MBedJJCNuOWXSvOt VZ7Q== MIME-Version: 1.0 X-Received: by 10.152.219.37 with SMTP id pl5mr11808779lac.36.1393238158061; Mon, 24 Feb 2014 02:35:58 -0800 (PST) Received: by 10.112.144.101 with HTTP; Mon, 24 Feb 2014 02:35:58 -0800 (PST) In-Reply-To: References: Date: Mon, 24 Feb 2014 11:35:58 +0100 Message-ID: Subject: Re: heap space error From: Dieter De Witte To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a113360447fb5ac04f32489cc X-Virus-Checked: Checked by ClamAV on apache.org --001a113360447fb5ac04f32489cc Content-Type: text/plain; charset=ISO-8859-1 No problem, it's not easy to learn about all hadoop's configuration options. Definitely consider looking into the reference (Tom White) 2014-02-24 11:20 GMT+01:00 Raj hadoop : > Thanks a ton Dieter > > > On Mon, Feb 24, 2014 at 3:45 PM, Dieter De Witte wrote: > >> You can configure the heap size of the mappers with the following >> parameter (in mapred.site.xml) >> >> mapred.map.child.java.opts=-Xmx3200m >> >> Also setting the nummber of map tasks is not useful. You should set the >> number of map slots per node: >> >> mapred.tasktracker.map.tasks.maximum=6 >> >> Regards, >> Dieter >> >> >> 2014-02-24 11:08 GMT+01:00 Raj hadoop : >> >> Hi All >>> >>> In our Map reduce code, when we are giving more >>> than 10 input sequence files, we are facing the java heap space error. >>> Kindly Find the attached screen shot for error and log file for failed >>> task. The program is working fine when number of input files is 10. >>> >>> I tried to set number of Map Tasks to 10 from code and also input to jar >>> but both are not working. >>> >>> Alternatives tried : >>> >>> 1. >>> >>> While running the map-reduce jar , providing the input as / -D >>> mapred.map.tasks =10. >>> 2. >>> >>> In code I changed Job to JobConf to set num of map-tasks as : >>> >>> JobConf job = new JobConf(conf, SplitAutomation.class); >>> >>> job.setNumMapTasks(10); >>> >>> Kindly help us as soon as possible as this is high priority. >>> >>> >>> >>> >>> >> > --001a113360447fb5ac04f32489cc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
No problem, it's not easy to learn about all hadoop= 9;s configuration options. Definitely consider looking into the reference (= Tom White)


2014-02-24 11:20 GMT+01:00 Raj hadoop <raj.hadoop@gmail.com>:
Thanks a ton Dieter


On Mo= n, Feb 24, 2014 at 3:45 PM, Dieter De Witte <drdwitte@gmail.com>= wrote:
You can configure= the heap size of the mappers with the following parameter (in mapred.site.= xml)

mapred.map.child.java.opts=3D-Xmx3200m

Also set= ting the nummber of map tasks is not useful. You should set the number of m= ap slots per node:

mapred.tasktracker.map.tasks.maximum=3D6

Regards,
= Dieter


2014-02-24 11:08 GMT+01:00 Raj hadoop <raj.hadoop@gmail.com>:

Hi All

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0=A0=A0 In our Map reduce code, when we are giving more than 10 input sequence files, we are facing the java heap space error. Kindly Find the attached screen shot for error and log file for fail= ed task. The program is working fine when number of input files is 10.<= /p>

I tried to set number of Map Tasks to 10 from code and also input to jar =A0but both are not working.

Alternatives tried :

  1. While running the map-reduce jar , providing the input as / -D mapred.map.tasks = =3D10.

  2. In code I changed Job to JobConf to set num of map-tasks as :

    JobConf job =3D new JobConf(conf, SplitAutomation.class);

    job.setNumMapTasks(10);

Kindly help us as soon as possible as this is high priority= .

=A0





--001a113360447fb5ac04f32489cc--