hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Marron <Peter.Mar...@trilliumsoftware.com>
Subject RE: libhdfs on windows
Date Wed, 31 Oct 2012 12:47:17 GMT
Hi Colin,

I have a requirement to be able to run all the Hadoop functionality
that I build from a variety of platforms. This was the original
motivation for wanting to use libhdfs. I followed your
suggestion and looked into using webhdfs and it's looking
promising. Thanks for that. However I also need to be able
to launch Map/Reduce jobs from any platform.
In particular from Windows.  I looked into this by hacking
the bin/hadoop  script to extract the required class path
and various arguments so that I could launch a Map/Reduce job
just by invoking java with the correct arguments.
However I ran into HADOOP-7682.
I can see that there is a workaround here
https://github.com/congainc/patch-hadoop_7682-1.0.x-win
but it suggests that this is not really appropriate for
deployment. I suspect that I can get it to work reliably
by using cygwin and making loads of modifications
but that all seems rather a large effort, error-prone and
difficult to maintain.

Given that I plan to have a relatively small repertoire of Map/Reduce
jobs that I need to launch, I'm tempted to have all the jars pre-packed
on the Name-Node and have the ability to run them there. Then
have a daemon running so that I can use any appropriate ad-hoc RPC
mechanism from Windows to launch them.

Am I missing something? Is there a way to launch Map/Reduce
jobs in a platform neutral way, which runs out of the box, on Windows?

Again, any suggestions welcome.

Peter Marron

> -----Original Message-----
> From: Peter Marron
> [mailto:Peter.Marron@trilliumsoftware.
> com]
> Sent: 26 October 2012 00:53
> To: common-dev@hadoop.apache.org
> Subject: RE: libhdfs on windows
> 
> Hi Colin,
> 
> OK, I didn't know there was a hdfs-dev.
> I'm happy to ask there.
> (However there's a lot of mail on
> dev@hadoop and user@hadoop as well
> as user@hive and it's a bit of a
> commitment to track them all.) As for
> webhdfs, I did think about that, and in
> some ways it's a beautiful solution as it
> gives me a platform- and language-
> neutral access mechanism. I was just a
> little worried about the HTTP overhead if
> I am reading a single record at a time.
> Also I will need some way to launch my
> Map/Reduce jobs as well. So I'll probably
> end up using the C++/JNI/Java route to
> do that anyway. Unless there's a better
> way?
> Is there a web Map/Reduce interface?
> 
> Many thanks,
> 
> Z
> 
> > -----Original Message-----
> > From: rarecactus@gmail.com
> > [mailto:rarecactus@gmail.com] On
> > Behalf Of Colin McCabe
> > Sent: 25 October 2012 18:24
> > To: common-dev@hadoop.apache.org
> > Subject: Re: libhdfs on windows
> >
> > Hi Peter,
> >
> > This might be a good question for hdfs-
> dev?
> >
> > As Harsh pointed out below, HDFS-573
> > was never committed.  I don't even see
> a patch attached, although
> > there is some discussion.
> >
> > In the mean time, might I suggest using
> the webhdfs interface on
> > Windows?
> > webhdfs was intended as a stable REST
> > interface that can be accessed from
> any platform.
> >
> > cheers,
> > Colin
> >
> >
> > On Thu, Oct 25, 2012 at 7:19 AM, Peter
> Marron
> > <Peter.Marron@trilliumsoftware.com>
> > wrote:
> > > Hi,
> > >
> > > I've been looking at using libhdfs and
> I
> > would like to use it on windows.
> > > I have found HDFS-573 and the
> > information on this page:
> > >
> >
> http://issues.apache.org/jira/browse/HD
> > FS-573?page=com.atlassian.jira.
> > > plugin.system.issuetabpanels:all-
> > tabpanel
> > > which suggests that quite a lot of
> work
> > was done on this way back in 2009.
> > > So is there some source from this
> > effort retained somewhere? If so,
> > where?
> > > Or do I have to start from scratch?
> > Apologies if this has already been asked
> recently.
> > >
> > > Any help appreciated.
> > >
> > > Peter Marron
> 
> 



Mime
View raw message