Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (nike.apache.org: domain of teddyyyy123@gmail.com
 designates 209.85.217.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAnh3__KYweRSYk5vzA8My_e1xmc9Oen636DJcxKRfSkPwuPhg@mail.gmail.com>
References: 
 <CAAnh3__KYweRSYk5vzA8My_e1xmc9Oen636DJcxKRfSkPwuPhg@mail.gmail.com>
From: Yang <teddyyyy123@gmail.com>
Date: Thu, 9 Oct 2014 11:40:48 -0700
Message-ID: 
 <CAAnh3_-LQaua6sHsNZ+PXjtq4LUKn8P=1kmdPe0YNoLE_9VLUw@mail.gmail.com>
Subject: Re: SSVD Q-Job taking very long even after 100% ?
To: user@mahout.apache.org
Content-Type: multipart/alternative; boundary=047d7b3a87f25b4232050501c757

--047d7b3a87f25b4232050501c757
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

it's possible that  they are compressing the output, I'm now rebuilding the
code after commenting out the setOutputCompress(true) in the code

also will run with compression param set to false


but still it's quite surprising why compression should take so long
(8--10minutes)

On Thu, Oct 9, 2014 at 11:06 AM, Yang <teddyyyy123@gmail.com> wrote:

> my Q-Job MR job shows as 100% mapper complete (it's a map-only job) very
> quickly, but the job itself does not finish, until about 10 minutes later=
.
> this is rather surprising. my input is a sparse vector of 37000 rows, and
> the column count is 8000, with each row usually having < 10 elements set =
to
> non-zero. so the input size is fairly small.
>
>
> I looked at the Q-job code, it seems rather normal, i.e. it's not doing
> anything special after the map() function is completed. so I wonder why
> it's lagging so long after 100% ?
>
>
> here is the syslog from hadoop:
>
>
>
> 2014-10-09 10:37:40,504 INFO [main] org.apache.hadoop.io.compress.zlib.Zl=
ibFactory: Successfully loaded & initialized native-zlib library
> 2014-10-09 10:37:40,538 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new decompressor [.gz]
> 2014-10-09 10:37:40,548 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new decompressor [.gz]
> 2014-10-09 10:37:40,548 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new decompressor [.gz]
> 2014-10-09 10:37:40,549 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new decompressor [.gz]
> 2014-10-09 10:39:39,143 WARN [communication thread] org.apache.hadoop.yar=
n.util.ProcfsBasedProcessTree: Error reading the stream java.io.IOException=
: No such process
> 2014-10-09 10:40:09,117 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new compressor [.deflate]
> 2014-10-09 10:46:23,991 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new decompressor [.deflate]
> 2014-10-09 10:46:23,992 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new decompressor [.deflate]
> 2014-10-09 10:46:23,992 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new decompressor [.deflate]
> 2014-10-09 10:46:23,992 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new decompressor [.deflate]
> 2014-10-09 10:46:31,219 INFO [LeaseRenewer:yyang15@apollo-phx-nn.vip.ebay=
.com:8020] org.apache.hadoop.ipc.Client: Retrying connect to server: apollo=
-phx-nn.vip.ebay.com/10.115.201.75:8020. Already tried 0 time(s); maxRetrie=
s=3D45
> 2014-10-09 10:47:45,241 INFO [main] org.apache.hadoop.io.compress.CodecPo=
ol: Got brand-new compressor [.deflate]
> 2014-10-09 10:47:46,571 INFO [main] org.apache.hadoop.mapred.Task: Task:a=
ttempt_1412781120464_7857_m_000000_0 is done. And is in the process of comm=
itting
> 2014-10-09 10:47:46,739 INFO [main] org.apache.hadoop.mapred.Task: Task a=
ttempt_1412781120464_7857_m_000000_0 is allowed to commit now
> 2014-10-09 10:47:47,389 INFO [main] org.apache.hadoop.mapreduce.lib.outpu=
t.FileOutputCommitter: Saved output of task 'attempt_1412781120464_7857_m_0=
00000_0' to hdfs://apollo-phx-nn.vip.ebay.com:8020/user/yyang15/CIReco/shoe=
s/ssvd/tmp/ssvd/Q-job/_temporary/1/task_1412781120464_7857_m_000000
> 2014-10-09 <http://apollo-phx-nn.vip.ebay.com:8020/user/yyang15/CIReco/sh=
oes/ssvd/tmp/ssvd/Q-job/_temporary/1/task_1412781120464_7857_m_0000002014-1=
0-09> 10:47:47,574 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt=
_1412781120464_7857_m_000000_0' done.
> 2014-10-09 10:47:47,575 INFO [main] org.apache.hadoop.metrics2.impl.Metri=
csSystemImpl: Stopping MapTask metrics system...
> 2014-10-09 10:47:47,576 INFO [ganglia] org.apache.hadoop.metrics2.impl.Me=
tricsSinkAdapter: ganglia thread interrupted.
> 2014-10-09 10:47:47,576 INFO [main] org.apache.hadoop.metrics2.impl.Metri=
csSystemImpl: MapTask metrics system stopped.
> 2014-10-09 10:47:47,576 INFO [main] org.apache.hadoop.metrics2.impl.Metri=
csSystemImpl: MapTask metrics system shutdown complete.
>
>

--047d7b3a87f25b4232050501c757--