hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dinkar Sitaram <dinkar.sita...@gmail.com>
Subject Re: Is hadoop tread safe?
Date Fri, 30 Aug 2013 11:54:58 GMT
This comment (from
http://stackoverflow.com/questions/12504690/how-to-run-hadoop-multithread-way-in-single-jvm
)
may also be relevant:

"Hadoop purposely does not run more than one task at the same time in one
JVM for isolation purposes. And in stand-alone (local) mode, only one JVM
is ever used. If you want to make use of your four cores, you should run in
pseudo-distributed mode, and increase the max number of concurrent tasks to
four. You can do this with the mapred.tasktracker.map.tasks.maximum and
mapred.tasktracker.reduce.tasks.maximum properties."


On Thu, Aug 29, 2013 at 8:49 PM, Harsh J <harsh@cloudera.com> wrote:

> Map tasks run in parallel spawned JVMs, so are isolated from one
> another at runtime. Use of static functions shouldn't affect you
> generally.
>
> Default Map I/O is single-threaded. If you plan to use
> multiple-threads, use MultiThreadedMapper for proper thread-safety.
>
> On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pavan0591@gmail.com>
> wrote:
> > No, I had written a huge Map Reduce program which talks with hbase and
> does
> > a lot of computing using it as a source as well as sink.. One of my
> > colleague saw my code and saw that I had used a lot of static function
> > instead of making use of proper OOP concepts.. He was telling me that it
> > shouldn't be the way I should go about doing it.. But my code works
> fine..
> > So, was wondering will I face any problem in the future because of this..
> > That's all.
> >
> > Regards,
> > Pavan
> >
> > On Aug 29, 2013 8:11 PM, "Adam Muise" <amuise@hortonworks.com> wrote:
> >>
> >> Mappers don't communicate with each other in traditional MapReduce. If
> you
> >> need something more MPI-ish then look to MPI over YARN or write your own
> >> YARN app.
> >>
> >> If you need multi-threading within the mapper then it is up to you as
> the
> >> java developer to make it thread safe. Use the concurrent libraries like
> >> anything else and Bob's your uncle. Having overly-complicated mappers
> can be
> >> difficult to manage however and it kind of misses the mark for MapReduce
> >> problems.
> >>
> >> Maybe if you expand on your use case a bit someone here can provide
> >> specific advice.
> >>
> >>
> >> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pavan0591@gmail.com
> >
> >> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
> >>> little bit of information on the way they execute in parallel would
> help me
> >>> out. Thanks.
> >>>
> >>> Regards,
> >>> Pavan
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Adam Muise
> >> Solution Engineer
> >> Hortonworks
> >> amuise@hortonworks.com
> >> 416-417-4037
> >>
> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
> Hadoop.
> >>
> >> Hortonworks Virtual Sandbox
> >>
> >> Hadoop: Disruptive Possibilities by Jeff Needham
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to which it is addressed and may contain information that is
> confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
>
>
>
> --
> Harsh J
>

Mime
View raw message