hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Is hadoop tread safe?
Date Thu, 29 Aug 2013 15:19:11 GMT
Map tasks run in parallel spawned JVMs, so are isolated from one
another at runtime. Use of static functions shouldn't affect you

Default Map I/O is single-threaded. If you plan to use
multiple-threads, use MultiThreadedMapper for proper thread-safety.

On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <pavan0591@gmail.com> wrote:
> No, I had written a huge Map Reduce program which talks with hbase and does
> a lot of computing using it as a source as well as sink.. One of my
> colleague saw my code and saw that I had used a lot of static function
> instead of making use of proper OOP concepts.. He was telling me that it
> shouldn't be the way I should go about doing it.. But my code works fine..
> So, was wondering will I face any problem in the future because of this..
> That's all.
> Regards,
> Pavan
> On Aug 29, 2013 8:11 PM, "Adam Muise" <amuise@hortonworks.com> wrote:
>> Mappers don't communicate with each other in traditional MapReduce. If you
>> need something more MPI-ish then look to MPI over YARN or write your own
>> YARN app.
>> If you need multi-threading within the mapper then it is up to you as the
>> java developer to make it thread safe. Use the concurrent libraries like
>> anything else and Bob's your uncle. Having overly-complicated mappers can be
>> difficult to manage however and it kind of misses the mark for MapReduce
>> problems.
>> Maybe if you expand on your use case a bit someone here can provide
>> specific advice.
>> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <pavan0591@gmail.com>
>> wrote:
>>> Hi all,
>>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>>> little bit of information on the way they execute in parallel would help me
>>> out. Thanks.
>>> Regards,
>>> Pavan
>> --
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise@hortonworks.com
>> 416-417-4037
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>> Hortonworks Virtual Sandbox
>> Hadoop: Disruptive Possibilities by Jeff Needham
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.

Harsh J

View raw message