hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KrzyCube <yuxh...@gmail.com>
Subject RE: Calling FsShell.doMain() hold so many threads
Date Tue, 24 Jul 2007 09:13:36 GMT

I have tried the way TestDFSShell.java does,
here's my code:

------------------------------------------------------------
public class CustomInterface 
{	
	Configuration conf ;
	FsShell fs ;
	
	public CustomInterface()
	{
		conf = new Configuration();
		fs = new FsShell();
		
		fs.setConf(conf);
	}

        public int createDir(String strDirName,String strPath)
	{
                // omit exception catch
		int iRet = 0;
	        strPath += strDirName;
	        String[] strCmd = new String[2];
		strCmd[0] = "-mkdir";
		strCmd[1] = strPath;		
		return m_fs.run(strCmd);		
	}	
}
------------------------------------------------------------

Then i just call the createdir Method

for(int i =0 ; i < 100000 ; i ++)
{
    custom.createDir("someName");
}

this cause the java vm process hold many threads
and these threads eat memory .
till the JVM Heap are eat up , throws Exceptions.
only larger Heap size holds more threads , but not fix the problem.

thanks.


Dhruba Borthakur wrote:
> 
> One example of programmatically using FsShell is in
> src/test/org/apache/hadoop/dfs/TestDFSShell.java
> 
> Thanks,
> dhruba
> 
> -----Original Message-----
> From: KrzyCube [mailto:yuxh312@gmail.com] 
> Sent: Monday, July 23, 2007 7:49 PM
> To: hadoop-user@lucene.apache.org
> Subject: Calling FsShell.doMain() hold so many threads
> 
> 
> Hi there:
> 
> i got two questions:
> 
> Q1:
>     I am try to  call the FsShell.doMain() with my own code , which is
> only
> a easy wrapper of the FsShell.
> But when i am trying to create many dirs , 10000 etc. Exception like "Not
> enough memory for more threads" throw ,  i have set the -Xmx512m.
>     Then i trying to view the process info while the program running ,
> then
> i found there are more and more threads invoked during the process , and
> eat
> more and more memory ,all threads still there without exit.
>     Then i came to the source code , and found that while the
> FsShell.Main()
> for terminal call there is one line "System.exit(return_value_of_doMain)"
> ,
> Is that mean the call of the ToolBase.run() which implemented in
> FsShell.java is always create a new thread and have to be force terminated
> by System.exit() to kill the process ?
>     So , if that is , how can i write my own code to use hadoop with
> FsShell
> in multi-thread mode , or is there any other way to do this ?
> 
> Q2:
>      I svn code  , and run it in eclipse [the only reason i refer to
> eclipse
> is to indicate my environment],
> under Unbuntu 7.04.
>      all about casual , i want to see how much time the FsShell.doMain()
> take , I use "new Date()" and 
> get the interval with "DateEnd.getTime() - DateBeg.getTime()"
>      Then i found that: even mkdir take more then 1000 [getTime shows]
> if there's no arguments , it take 25 , but even if i just give it a wrong
> argument , such as "-sl", it take more than 1000 , is that means the
> argument check take most of the time cost?
> 
> -- 
> View this message in context:
> http://www.nabble.com/Calling-FsShell.doMain%28%29-hold-so-many-threads-tf41
> 33557.html#a11756139
> Sent from the Hadoop Users mailing list archive at Nabble.com.
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Calling-FsShell.doMain%28%29-hold-so-many-threads-tf4133557.html#a11759406
Sent from the Hadoop Users mailing list archive at Nabble.com.


Mime
View raw message