hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralph Castain <...@open-mpi.org>
Subject Re: Help with error
Date Mon, 09 Apr 2012 17:45:44 GMT
Hi Bobby

On Apr 9, 2012, at 11:40 AM, Robert Evans wrote:

> What do you mean by relocated some supporting files to HDFS?  How do you relocate them?
 What API do you use?

I use the LocalResource and FileSystem classes to do the relocation, per the Hadoop example:

	// set local resources for the application master
	// local files or archives as needed
	// In this scenario, the jar file for the application master is part of the local resources
		
	Map<String, LocalResource> localResources = new HashMap<String, LocalResource>();

	LOG.info("Copy openmpi tarball from local filesystem and add to local environment");
	// Copy the application master jar to the filesystem 
	// Create a local resource to point to the destination jar path 
	FileSystem fs;
	FileStatus destStatus;
	try {
	    fs = FileSystem.get(conf);
	    Path src = new Path(pathOMPItarball);
	    String pathSuffix = appName + "/" + appId.getId();	    
	    Path dst = new Path(fs.getHomeDirectory(), pathSuffix);
	    try {
		fs.copyFromLocalFile(false, true, src, dst);
		try {
		    destStatus = fs.getFileStatus(dst);
		    LocalResource amJarRsrc = Records.newRecord(LocalResource.class);
	
		    // Set the type of resource - file or archive
		    // archives are untarred at destination
		    amJarRsrc.setType(LocalResourceType.ARCHIVE);
		    // Set visibility of the resource 
		    // Setting to most private option
		    amJarRsrc.setVisibility(LocalResourceVisibility.APPLICATION);	   
		    // Set the resource to be copied over
		    amJarRsrc.setResource(ConverterUtils.getYarnUrlFromPath(dst)); 
		    // Set timestamp and length of file so that the framework 
		    // can do basic sanity checks for the local resource 
		    // after it has been copied over to ensure it is the same 
		    // resource the client intended to use with the application
		    amJarRsrc.setTimestamp(destStatus.getModificationTime());
		    amJarRsrc.setSize(destStatus.getLen());
		    localResources.put("openmpi",  amJarRsrc);
		} catch (Throwable t) {
		    LOG.fatal("Error on file status", t);
		    System.exit(1);
		}
	    } catch (Throwable t) {
		LOG.fatal("Error on copy from local file", t);
		System.exit(1);
	    }
	} catch (Throwable t) {
	    LOG.fatal("Error getting filesystem configuration", t);
	    System.exit(1);
	}

Note that this appears to work fine when the local resource type was "file" - at least, I
was able to make a simple program work that way. Problem I'm having is when I move an archive,
which is why I was hoping to look at the HDFS end to see what files are present, and in what
locations so I can set the paths accordingly.

Thanks
Ralph


> 
> --Bobby Evans
> 
> 
> On 4/9/12 11:10 AM, "Ralph Castain" <rhc@open-mpi.org> wrote:
> 
> Hi folks
> 
> I'm trying to develop an AM for the 0.23 branch and running into a problem that I'm having
difficulty debugging. My client relocates some supporting files to HDFS, creates the application
object for the AM, and submits it to the RM.
> 
> The file relocation request doesn't generate an error, so I must assume it succeeded.
It would be nice if there was some obvious way to verify that, but I haven't discovered it.
Can anyone give me a hint? I tried asking hdfs to -ls, but all I get is that "." doesn't exist.
I have no idea where the file would be placed, if it would persist once the job fails, etc.
> 
> When the job is submitted, all I get is an "Error 500", which tells me nothing. Reminds
me of the old days of 40 years ago when you'd get the dreaded "error 11", which meant anything
from a divide by zero to a memory violation. Are there any debug flags I could set that might
provide more info?
> 
> Thanks
> Ralph
> 
> 


Mime
View raw message