Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AC4BCDBAE for ; Tue, 6 Nov 2012 16:28:20 +0000 (UTC) Received: (qmail 57222 invoked by uid 500); 6 Nov 2012 16:28:15 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 57039 invoked by uid 500); 6 Nov 2012 16:28:15 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 57019 invoked by uid 99); 6 Nov 2012 16:28:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Nov 2012 16:28:14 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com designates 65.55.111.105 as permitted sender) Received: from [65.55.111.105] (HELO blu0-omc2-s30.blu0.hotmail.com) (65.55.111.105) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Nov 2012 16:28:04 +0000 Received: from BLU0-SMTP58 ([65.55.111.73]) by blu0-omc2-s30.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Tue, 6 Nov 2012 08:27:43 -0800 X-Originating-IP: [173.15.87.37] X-EIP: [Bm2Me+QtphgzDK6BpO/F2OCtbs0xOO+g] X-Originating-Email: [michael_segel@hotmail.com] Message-ID: Received: from [192.168.0.100] ([173.15.87.37]) by BLU0-SMTP58.blu0.hotmail.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Tue, 6 Nov 2012 08:27:41 -0800 From: Michael Segel Content-Type: multipart/alternative; boundary="Apple-Mail=_3A928F0F-C7A9-4660-BAE4-A3B2C624194F" MIME-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: One mapper/reducer runs on a single JVM Date: Tue, 6 Nov 2012 10:27:45 -0600 References: To: user@hadoop.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-OriginalArrivalTime: 06 Nov 2012 16:27:42.0080 (UTC) FILETIME=[A4FDC800:01CDBC3B] X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_3A928F0F-C7A9-4660-BAE4-A3B2C624194F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1" If you exceed the amount of physical memory available, memory pages will = be written to disk in a temp space. The act of 'swapping' the memory = pages from memory to disk and back again is known as 'swap'.=20 HBase is highly sensitive to the latency of swapping memory in and out = of physical memory to disk. You need to avoid swap when running HBase. = It will crash a region server and ultimately you can end up with a = cascading failure and HBase will go down.=20 HTH -Mike On Nov 5, 2012, at 11:06 PM, Lin Ma wrote: > Thanks Michael, >=20 > "If you are running just Hadoop, you could have a little swap. Running = HBase, fuggit about it." -- could you give a bit more information about = what do you mean swap and why forget for HBase? >=20 > regards, > Lin >=20 >=20 > On Tue, Nov 6, 2012 at 12:46 PM, Michael Segel = wrote: > Mappers and Reducers are separate JVM processes. > And yes you need to take in to account the amount of memory the = machine(s) when you configure the number of slots. >=20 > If you are running just Hadoop, you could have a little swap. Running = HBase, fuggit about it. >=20 >=20 > On Nov 5, 2012, at 7:12 PM, Lin Ma wrote: >=20 > > Hello Hadoop experts, > > > > I have a question in my mind for a long time. Supposing I am = developing M-R program, and it is Java based (Java UDF, implements = mapper or reducer interface). My question is, in this scenario, whether = a mapper or a reducer is a separate JVM process? E.g. supposing on a = machine, there are 4 mappers, they are 4 individual processes? I am also = wondering whether the processes on a single machine will impact each = other when each JVM wants to get more memory to run faster? > > > > thanks in advance, > > Lin > > > > >=20 >=20 --Apple-Mail=_3A928F0F-C7A9-4660-BAE4-A3B2C624194F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="iso-8859-1" If = you exceed the amount of physical memory available, memory pages will be = written to disk in a temp space. The act of 'swapping' the memory pages = from memory to disk and back again is known as = 'swap'. 

HBase is highly sensitive to the = latency of swapping memory in and out of physical memory to disk. You = need to avoid swap when running HBase.  It will crash a region = server and ultimately you can end up with a cascading failure and HBase = will go = down. 

HTH

-Mike

On Nov 5, 2012, at 11:06 PM, Lin Ma <linlma@gmail.com> = wrote:

Thanks Michael,

"If you are running just Hadoop, = you could have a little swap. Running HBase, fuggit about it." -- could = you give a bit more information about what do you mean swap and why = forget for HBase?

regards,
Lin


On Tue, Nov 6, 2012 at 12:46 PM, Michael Segel = <michael_segel@hotmail.com> wrote:
Mappers and Reducers = are separate JVM processes.
And yes you need to take in to account the amount of memory the = machine(s) when you configure the number of slots.

If you are running just Hadoop, you could have a little swap. Running = HBase, fuggit about it.


On Nov 5, 2012, at 7:12 PM, Lin Ma <linlma@gmail.com> wrote:

> Hello Hadoop experts,
>
> I have a question in my mind for a long time. Supposing I am = developing M-R program, and it is Java based (Java UDF, implements = mapper or reducer interface). My question is, in this scenario, whether = a mapper or a reducer is a separate JVM process? E.g. supposing on a = machine, there are 4 mappers, they are 4 individual processes? I am also = wondering whether the processes on a single machine will impact each = other when each JVM wants to get more memory to run faster?
>
> thanks in advance,
> Lin
>
>



= --Apple-Mail=_3A928F0F-C7A9-4660-BAE4-A3B2C624194F--