Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 56B0B109D2 for ; Thu, 29 Aug 2013 14:41:43 +0000 (UTC) Received: (qmail 74683 invoked by uid 500); 29 Aug 2013 14:41:34 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 74590 invoked by uid 500); 29 Aug 2013 14:41:27 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 74388 invoked by uid 99); 29 Aug 2013 14:41:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Aug 2013 14:41:25 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of amuise@hortonworks.com designates 74.125.82.176 as permitted sender) Received: from [74.125.82.176] (HELO mail-we0-f176.google.com) (74.125.82.176) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Aug 2013 14:41:19 +0000 Received: by mail-we0-f176.google.com with SMTP id q56so506874wes.21 for ; Thu, 29 Aug 2013 07:40:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=2GTojBJeavl44NhSbwKuq4Vr5lt2WImsI78sjzYXBnU=; b=lwiI7p2HoQjsuhHqMXo/n6pMlHnvuY/KWFzTPbczVxBOt1o3NvMmmVtqVmb6/owvYD +BHesuHsG6jAkNUoSU96YAwWqHwcS+AHqUIndXg4kDyoxpRznGJg2Fx3LQvOD9e7Kvs8 VO2xOgHqi9a1ORWagV0hzEeba+AgaKUGNJXqfJQJLMyw8PXTRhpbM5RgXnS6voxjhyQl E/iXxvWW2zMakK4MDNv8+uH1sWtQwg4ahXje+1rGHoBR3bdxri9xCjv4cux2zDDMbuNz CvOmCBoJlY/uin+JH2iPMfzr2zYYWiOCWof43iOB2FOJK5td6N4Q0sHtGRhFBHhEqccE 9RVw== X-Gm-Message-State: ALoCoQmyYJCW1XRa+GfnCHZOnA3aV6Wyqalqz5DI+pw4LFZJLiZFOO3sW5N1CRzgDTfIH2ze8yJjO+3CW0m5SGzKHFsTF7bDVc4Uvps4JzHdU8zahu6m2RU= MIME-Version: 1.0 X-Received: by 10.180.206.180 with SMTP id lp20mr15146945wic.48.1377787259008; Thu, 29 Aug 2013 07:40:59 -0700 (PDT) Received: by 10.216.139.71 with HTTP; Thu, 29 Aug 2013 07:40:58 -0700 (PDT) In-Reply-To: References: Date: Thu, 29 Aug 2013 10:40:58 -0400 Message-ID: Subject: Re: Is hadoop tread safe? From: Adam Muise To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c380b0263e4c04e5171809 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c380b0263e4c04e5171809 Content-Type: text/plain; charset=US-ASCII Mappers don't communicate with each other in traditional MapReduce. If you need something more MPI-ish then look to MPI over YARN or write your own YARN app. If you need multi-threading within the mapper then it is up to you as the java developer to make it thread safe. Use the concurrent libraries like anything else and Bob's your uncle. Having overly-complicated mappers can be difficult to manage however and it kind of misses the mark for MapReduce problems. Maybe if you expand on your use case a bit someone here can provide specific advice. On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra wrote: > Hi all, > > Is hadoop thread safe? Do mappers make use of threads in any chance? A > little bit of information on the way they execute in parallel would help me > out. Thanks. > > Regards, > Pavan > -- * * * * *Adam Muise* Solution Engineer *Hortonworks* amuise@hortonworks.com 416-417-4037 Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop. Hortonworks Virtual Sandbox Hadoop: Disruptive Possibilities by Jeff Needham -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --001a11c380b0263e4c04e5171809 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Mappers don't communicate with each other in tradition= al MapReduce. If you need something more MPI-ish then look to MPI over YARN= or write your own YARN app.

If you need multi-thr= eading within the mapper then it is up to you as the java developer to make= it thread safe. Use the concurrent libraries like anything else and Bob= 9;s your uncle. Having overly-complicated mappers can be difficult to manag= e however and it kind of misses the mark for MapReduce problems.=A0

Maybe if you expand on your use case a bit = someone here can provide specific advice.


On Thu, Aug 29, 2013 at 10:33 AM, P= avan Sudheendra <pavan0591@gmail.com> wrote:

Hi all,

Is hadoop thread safe? Do mappers make use of threads in any= chance? A little bit of information on the way they execute in parallel wo= uld help me out. Thanks.

Regards,
Pavan




--

CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --001a11c380b0263e4c04e5171809--