hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tareq Aljabban <dee.jay23...@gmail.com>
Subject Re: libhdfs process fork problem
Date Tue, 27 Mar 2012 15:07:36 GMT
Hi Brian,
Thanks for your response..
Actually the master process doesn't even try to access HDFS..
What's happening is that process Master forks 2 processes A and B.
A and B are the ones that try to connect to HDFS..
I tried doing this in a separate application and it worked.. for some
reason it doesn't work in the other application that I'm working on.
Since I'm not connecting from master then I cannot really apply any of your
advices.. do you have other ideas?

On Fri, Mar 23, 2012 at 7:25 AM, Brian Bockelman <bbockelm@cse.unl.edu>wrote:

> Hi Tareq,
> This is because libhdfs will keep a bit of state data (especially if the
> master was connected to HDFS).
> Three suggestions:
> 1) [Likely to work] Fork the children first, then do any HDFS actions in
> the master.  Alternately, don't have the master do any HDFS actions; have
> it fork a child which does them for it.
> 2) [May work] Try disconnecting the master from HDFS, then forking the
> children.
> 3) [If 2 doesn't work] Have the master disconnect, then fork the children,
> then have the children exec.
> Brian
> On Mar 22, 2012, at 11:38 PM, Tareq Aljabban wrote:
> > Hi,
> > I'm using libhdfs C interface to access HDFS.
> > The application accessing HDFS has a master process that forks many
> > processes, and these processes try to connect to HDFS.
> > Upon connection, one of the processes is exiting as soon as it reaches
> the
> > hdfsConnect().
> > I tried simulating this behavior by creating a separate application where
> > the master process forks two processes that connect to HDFS and send
> > requests.. this worked without any problem.. so I got confused.. what's
> > really causing the connection problems then?
> > Any insights on this is much appreciated.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message