hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Milind.Bhandar...@emc.com>
Subject Re: Help with error
Date Mon, 09 Apr 2012 18:01:44 GMT
Ralph,

Do you have a home directory on HDFS ?

The job files are moved under your HDFS home directory, in a .staging
subdir.

In any case, not flagging it as an error, seems to be a genuine yarn bug.

- milind

---
Milind Bhandarkar
Greenplum Labs, EMC
(Disclaimer: Opinions expressed in this email are those of the author, and
do not necessarily represent the views of any organization, past or
present, the author might be affiliated with.)



On 4/9/12 10:45 AM, "Ralph Castain" <rhc@open-mpi.org> wrote:

>Hi Bobby
>
>On Apr 9, 2012, at 11:40 AM, Robert Evans wrote:
>
>> What do you mean by relocated some supporting files to HDFS?  How do
>>you relocate them?  What API do you use?
>
>I use the LocalResource and FileSystem classes to do the relocation, per
>the Hadoop example:
>
>	// set local resources for the application master
>	// local files or archives as needed
>	// In this scenario, the jar file for the application master is part of
>the local resources			
>	Map<String, LocalResource> localResources = new HashMap<String,
>LocalResource>();
>
>	LOG.info("Copy openmpi tarball from local filesystem and add to local
>environment");
>	// Copy the application master jar to the filesystem
>	// Create a local resource to point to the destination jar path
>	FileSystem fs;
>	FileStatus destStatus;
>	try {
>	    fs = FileSystem.get(conf);
>	    Path src = new Path(pathOMPItarball);
>	    String pathSuffix = appName + "/" + appId.getId();	
>	    Path dst = new Path(fs.getHomeDirectory(), pathSuffix);
>	    try {
>		fs.copyFromLocalFile(false, true, src, dst);
>		try {
>		    destStatus = fs.getFileStatus(dst);
>		    LocalResource amJarRsrc = Records.newRecord(LocalResource.class);
>	
>		    // Set the type of resource - file or archive
>		    // archives are untarred at destination
>		    amJarRsrc.setType(LocalResourceType.ARCHIVE);
>		    // Set visibility of the resource
>		    // Setting to most private option
>		    amJarRsrc.setVisibility(LocalResourceVisibility.APPLICATION);	
>		    // Set the resource to be copied over
>		    amJarRsrc.setResource(ConverterUtils.getYarnUrlFromPath(dst));
>		    // Set timestamp and length of file so that the framework
>		    // can do basic sanity checks for the local resource
>		    // after it has been copied over to ensure it is the same
>		    // resource the client intended to use with the application
>		    amJarRsrc.setTimestamp(destStatus.getModificationTime());
>		    amJarRsrc.setSize(destStatus.getLen());
>		    localResources.put("openmpi",  amJarRsrc);
>		} catch (Throwable t) {
>		    LOG.fatal("Error on file status", t);
>		    System.exit(1);
>		}
>	    } catch (Throwable t) {
>		LOG.fatal("Error on copy from local file", t);
>		System.exit(1);
>	    }
>	} catch (Throwable t) {
>	    LOG.fatal("Error getting filesystem configuration", t);
>	    System.exit(1);
>	}
>
>Note that this appears to work fine when the local resource type was
>"file" - at least, I was able to make a simple program work that way.
>Problem I'm having is when I move an archive, which is why I was hoping
>to look at the HDFS end to see what files are present, and in what
>locations so I can set the paths accordingly.
>
>Thanks
>Ralph
>
>
>> 
>> --Bobby Evans
>> 
>> 
>> On 4/9/12 11:10 AM, "Ralph Castain" <rhc@open-mpi.org> wrote:
>> 
>> Hi folks
>> 
>> I'm trying to develop an AM for the 0.23 branch and running into a
>>problem that I'm having difficulty debugging. My client relocates some
>>supporting files to HDFS, creates the application object for the AM, and
>>submits it to the RM.
>> 
>> The file relocation request doesn't generate an error, so I must assume
>>it succeeded. It would be nice if there was some obvious way to verify
>>that, but I haven't discovered it. Can anyone give me a hint? I tried
>>asking hdfs to -ls, but all I get is that "." doesn't exist. I have no
>>idea where the file would be placed, if it would persist once the job
>>fails, etc.
>> 
>> When the job is submitted, all I get is an "Error 500", which tells me
>>nothing. Reminds me of the old days of 40 years ago when you'd get the
>>dreaded "error 11", which meant anything from a divide by zero to a
>>memory violation. Are there any debug flags I could set that might
>>provide more info?
>> 
>> Thanks
>> Ralph
>> 
>> 
>
>


Mime
View raw message