mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy Lyubimov (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (MAHOUT-814) SSVD local tests should use their own tmp space to avoid collisions
Date Tue, 20 Sep 2011 02:38:09 GMT

    [ https://issues.apache.org/jira/browse/MAHOUT-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108328#comment-13108328
] 

Dmitriy Lyubimov edited comment on MAHOUT-814 at 9/20/11 2:37 AM:
------------------------------------------------------------------

another workaround, per tutorial, seems to be to construct this task per the rule below but
does it have to be that complicated just to access task scratchpad path? besides, i looked
at the history of this and it really subject to change. So using java.io.tmpdir really seems
like the safest bet. 

{quote}

$${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/work/tmp : The temporary directory
for the task. (User can specify the property mapred.child.tmp to set the value of temporary
directory for map and reduce tasks. This defaults to ./tmp. If the value is not an absolute
path, it is prepended with task's working directory. Otherwise, it is directly assigned. The
directory will be created if it doesn't exist. Then, the child java tasks are executed with
option -Djava.io.tmpdir='the absolute path of the tmp dir'. Anp pipes and streaming are set
with environment variable, TMPDIR='the absolute path of the tmp dir'). This directory is created,
if mapred.child.tmp has the value ./tmp


{quote}

      was (Author: dlyubimov):
    There's also some info floating around that job.local.dir property may be a better candidate
for task's scratch space than java.io.tmp.


  
> SSVD local tests should use their own tmp space to avoid collisions
> -------------------------------------------------------------------
>
>                 Key: MAHOUT-814
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-814
>             Project: Mahout
>          Issue Type: Bug
>    Affects Versions: 0.5
>            Reporter: Grant Ingersoll
>            Assignee: Dmitriy Lyubimov
>            Priority: Minor
>             Fix For: 0.6
>
>         Attachments: MAHOUT-814.patch
>
>
> Running Mahout in an environment with Jenkins also running and am getting:
> {quote}
> java.io.FileNotFoundException: /tmp/q-temp.seq (Permission denied)
>         at java.io.FileOutputStream.open(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:187)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:183)
>         at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:241)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
>         at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:368)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:528)
>         at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1198)
>         at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:401)
>         at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:284)
>         at org.apache.mahout.math.hadoop.stochasticsvd.qr.QRFirstStep.getTempQw(QRFirstStep.java:263)
>         at org.apache.mahout.math.hadoop.stochasticsvd.qr.QRFirstStep.flushSolver(QRFirstStep.java:104)
>         at org.apache.mahout.math.hadoop.stochasticsvd.qr.QRFirstStep.map(QRFirstStep.java:175)
>         at org.apache.mahout.math.hadoop.stochasticsvd.qr.QRFirstStep.collect(QRFirstStep.java:279)
>         at org.apache.mahout.math.hadoop.stochasticsvd.QJob$QMapper.map(QJob.java:142)
>         at org.apache.mahout.math.hadoop.stochasticsvd.QJob$QMapper.map(QJob.java:71)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
> {quote}
> Also seeing the following tests fail:
> {quote}
> Tests in error: 
>   testSSVDSolverSparse(org.apache.mahout.math.hadoop.stochasticsvd.LocalSSVDSolverSparseSequentialTest):
Q job unsuccessful.
>   testSSVDSolverPowerIterations1(org.apache.mahout.math.hadoop.stochasticsvd.LocalSSVDSolverSparseSequentialTest):
Q job unsuccessful.
>   testSSVDSolverPowerIterations1(org.apache.mahout.math.hadoop.stochasticsvd.LocalSSVDSolverDenseTest):
Q job unsuccessful.
>   testSSVDSolverDense(org.apache.mahout.math.hadoop.stochasticsvd.LocalSSVDSolverDenseTest):
Q job unsuccessful.
> {quote}
> I haven't checked all of them, but I suspect they are all due to the same reason.  We
should dynamically create a temp area for each test using temporary directories under the
main temp dir.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message