hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Silvina CaĆ­no Lores <silvi.ca...@gmail.com>
Subject Re: Drawbacks of Hadoop Pipes
Date Tue, 04 Mar 2014 09:24:48 GMT
Hi there,

I've been working with pipes for some months and I've finally managed to
get it working as I wanted with some legacy code I had. However, I had many
many issues regarding not only my implementation (it had to be adapted in
several ways to fit pipes, it is very restrictive) but pipes itself (bugs,
obscure errors and lack of proper logging with the subsequent mad

I also tried streaming but I found it even more complex to debug and I
found some deal-breaker errors that I couldn't overcome regarding buffering
and such. I also tried a SWIG interface to wrap my code into a Java
library, I'd never recommend that for you might end up introducing a lot of
memory issues and potential bugs into your already working code, and you
basically don't get anything useful from it.

I've never worked with CUDA though, but it shouldn't be any different from
my Hadoop Pipes deployment besides the specific libraries you need. Be
prepared to deal with configuration issues and many esoteric logs,

My advise, based in my experience, is that you should be 99% sure that your
original code is solid before migrating to Hadoop Pipes, you will have
enough problems there anyway.

Good luck on your work :)

On 3 March 2014 16:11, Basu,Indrashish <indrashish@ufl.edu> wrote:

> Hello,
> Anyone can help regarding the below query.
> Regards,
> Indrashish
> On Sat, 01 Mar 2014 13:52:11 -0500, Basu,Indrashish wrote:
>> Hello,
>> I am trying to execute a CUDA benchmark in a Hadoop Framework and
>> using Hadoop Pipes for invoking the CUDA code which is written in a
>> C++ interface from the Hadoop Framework. I am just a bit interested in
>> knowing what can be the drawbacks of using Hadoop Pipes for this and
>> whether the implementation of Hadoop Streaming and JNI interface will
>> be a better choice. I am a bit unclear on this, so if anyone can throw
>> some light on this and clarify.
>> Regards,
>> Indrashish
> --
> Indrashish Basu
> Graduate Student
> Department of Electrical and Computer Engineering
> University of Florida

View raw message