hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Earl <charles.ce...@gmail.com>
Subject Re: Streaming Hadoop using C
Date Wed, 29 Feb 2012 19:35:51 GMT
So if I understand, it is more the memory management that you are interested in, rather than
a need to run an existing C or C++ application in MapReduce platform?
Have you done profiling of the application?
On Feb 29, 2012, at 2:19 PM, Mark question wrote:

> Thanks Charles .. I'm running Hadoop for research to perform duplicate
> detection methods. To go deeper, I need to understand what's slowing my
> program, which usually starts with analyzing memory to predict best input
> size for map task. So you're saying piping can help me control memory even
> though it's running on VM eventually?
> Thanks,
> Mark
> On Wed, Feb 29, 2012 at 11:03 AM, Charles Earl <charles.cearl@gmail.com>wrote:
>> Mark,
>> Both streaming and pipes allow this, perhaps more so pipes at the level of
>> the mapreduce task. Can you provide more details on the application?
>> On Feb 29, 2012, at 1:56 PM, Mark question wrote:
>>> Hi guys, thought I should ask this before I use it ... will using C over
>>> Hadoop give me the usual C memory management? For example, malloc() ,
>>> sizeof() ? My guess is no since this all will eventually be turned into
>>> bytecode, but I need more control on memory which obviously is hard for
>> me
>>> to do with Java.
>>> Let me know of any advantages you know about streaming in C over hadoop.
>>> Thank you,
>>> Mark

View raw message