hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Gupta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5354) some job submission fail when one of the configured local disks on jt is read only
Date Tue, 25 Jun 2013 23:50:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693508#comment-13693508
] 

Arpit Gupta commented on MAPREDUCE-5354:
----------------------------------------

So some job submissions will go through and some will fail based on which ones end up using
the read only disk. It would be better if jobtracker could detect this bad disk and ignore
it for future use. Otherwise users will have remove this dir from the config and restart jobtracker
and thus loose all the jobs in progress or if they have recovery enabled they will start from
scratch.
                
> some job submission fail when one of the configured local disks on jt is read only
> ----------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5354
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5354
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 1.3.0
>            Reporter: Arpit Gupta
>
> Following exception is seen in the jt
> {code}
> 2013-06-25 06:58:26,988 INFO org.apache.hadoop.mapred.JobTracker: jobToken generated
and stored with users keys in /mapred/system/job_201306250645_0002/jobToken
> 2013-06-25 06:58:26,997 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:user cause:java.io.IOException: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml
(Read-only file system)
> 2013-06-25 06:58:26,997 INFO org.apache.hadoop.ipc.Server: IPC Server handler 33 on 50300,
call submitJob(job_201306250645_0002, hdfs://hostname:8020/user/hrt_qa/.staging/job_201306250645_0002,
org.apache.hadoop.security.Credentials@34e77781) from 68.142.244.30:56631: error: java.io.IOException:
java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml
(Read-only file system)
> java.io.IOException: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml
(Read-only file system)
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3615)
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1440)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1438)
> Caused by: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml
(Read-only file system)
>         at java.io.FileOutputStream.open(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:193)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:189)
>         at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:260)
>         at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:241)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
>         at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
>         at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:229)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1248)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1229)
>         at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:411)
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3613)
>         ... 12 more
> 2013-06-25 06:58:27,079 INFO org.apache.hadoop.mapred.JobInitializationPoller: Passing
to Initializer Job Id :job_2013062506
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message