hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: libhdfs process fork problem
Date Fri, 23 Mar 2012 11:25:32 GMT
Hi Tareq,

This is because libhdfs will keep a bit of state data (especially if the master was connected
to HDFS).

Three suggestions:
1) [Likely to work] Fork the children first, then do any HDFS actions in the master.  Alternately,
don't have the master do any HDFS actions; have it fork a child which does them for it.
2) [May work] Try disconnecting the master from HDFS, then forking the children.
3) [If 2 doesn't work] Have the master disconnect, then fork the children, then have the children
exec.

Brian

On Mar 22, 2012, at 11:38 PM, Tareq Aljabban wrote:

> Hi,
> I'm using libhdfs C interface to access HDFS.
> The application accessing HDFS has a master process that forks many
> processes, and these processes try to connect to HDFS.
> Upon connection, one of the processes is exiting as soon as it reaches the
> hdfsConnect().
> I tried simulating this behavior by creating a separate application where
> the master process forks two processes that connect to HDFS and send
> requests.. this worked without any problem.. so I got confused.. what's
> really causing the connection problems then?
> Any insights on this is much appreciated.


Mime
View raw message