Mailing-List: contact dev-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mahout.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1990.1\))
Subject: Re: AtA error
From: Pat Ferrel <pat@occamsmachete.com>
In-Reply-To: 
 <CAPud8Tq0muOZ8Ng7Gr2vDVGhLGzfwurEZsZ_1xdugxLdgLK62A@mail.gmail.com>
Date: Mon, 17 Nov 2014 17:24:24 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <51DD88CE-9FFD-4AC9-B285-1DB887C7A8EE@occamsmachete.com>
References: <9FECAC7D-E1DC-4E5D-97C4-3A72B94F4813@occamsmachete.com>
 <CAPud8TqcozW3xY2gXd4qUMHxo9Z5Z+knBBOBaqJ+d-LQebH6bQ@mail.gmail.com>
 <CAPud8TpJ28AVnobBjbd=RoWfOJ0ZaEp9R3KsZ3Py-iFUOLks3w@mail.gmail.com>
 <BAA0AE8E-C1C5-4187-849D-8F0DEAF7D345@occamsmachete.com>
 <CAPud8Tq0muOZ8Ng7Gr2vDVGhLGzfwurEZsZ_1xdugxLdgLK62A@mail.gmail.com>
To: dev@mahout.apache.org

I do use drmWrap so I=E2=80=99ll check there, thanks

On Nov 17, 2014, at 5:22 PM, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:

On Mon, Nov 17, 2014 at 5:16 PM, Pat Ferrel <pat@occamsmachete.com> =
wrote:

> It=E2=80=99s in spark-itemsimilarity. This job reads elements and =
assigns them to
> one of two RDD backed drms.
>=20
> I assumed it was a badly formed drm but it=E2=80=99s a 140MB dataset =
and a bit
> hard to nail down=E2=80=94just looking for a clue. I read this to say =
that an ID
> for an element in a row vector was larger than drm.ncol, correct?
>=20

yes.

and then it again comes back to the question how the matrix was
constructed. General construction of dimensions (ncol, nrow) is
automatic-lazy, meaning if you have not specified dimensions anywhere
explicitly, it will lazily compute it for you. But if you did volunteer
them anywhere (such as to drmWrap() call) they got to be good. Or you =
see
things like this.

>=20
>=20
> On Nov 17, 2014, at 4:58 PM, Dmitriy Lyubimov <dlieu.7@gmail.com> =
wrote:
>=20
> So this is not a problem of A'A computation -- the input is obviously
> invalid.
>=20
> Question is what you did before you got a A handle -- read it from =
file?
> parallelized it from in-core matrix (drmParallelize)? as a result of =
other
> computation (if yes than what)? wrapped around manually crafted RDD
> (drmWrap)?
>=20
> I don't understand the question about non-continuous ids. You are =
referring
> to some context of your computation assuming I am in context (but i am
> unfortunately not)
>=20
> On Mon, Nov 17, 2014 at 4:55 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> wrote:
>=20
>>=20
>>=20
>> On Mon, Nov 17, 2014 at 3:46 PM, Pat Ferrel <pat@occamsmachete.com>
> wrote:
>>=20
>>> A matrix with about 4600 rows and somewhere around 27790 columns =
when
>>> executing the following line from AtA (not sure of the exact =
dimensions)
>>>=20
>>>    /** The version of A'A that does not use GraphX */
>>>    def at_a_nongraph(op: OpAtA[_], srcRdd: DrmRdd[_]): DrmRdd[Int] =3D=
 {
>>>=20
>>> a vector is created whose size is causes the error. How could I have
>>> constructed a drm that would cause this error? If the column IDs =
were
>>> non-contiguous would that yield this error?
>>>=20
>>=20
>> what did you do specifically to build matrix A?
>>=20
>>=20
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>=20
>>> 14/11/12 17:56:03 ERROR executor.Executor: Exception in task 5.0 in
> stage
>>> 18.0 (TID 66169)
>>> org.apache.mahout.math.IndexException: Index 27792 is outside =
allowable
>>> range of [0,27789)
>>>       at
>>> =
org.apache.mahout.math.AbstractVector.viewPart(AbstractVector.java:147)
>>>       at
>>> =
org.apache.mahout.math.scalabindings.VectorOps.apply(VectorOps.scala:37)
>>>       at
>>>=20
> =
org.apache.mahout.sparkbindings.blas.AtA$$anonfun$5$$anonfun$apply$6.apply=
(AtA.scala:152)
>>>       at
>>>=20
> =
org.apache.mahout.sparkbindings.blas.AtA$$anonfun$5$$anonfun$apply$6.apply=
(AtA.scala:149)
>>>       at
>>> =
scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
>>>       at
>>> =
scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
>>>       at =
scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
>>>       at =
scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
>>>       at
>>>=20
> =
scala.collection.immutable.StreamIterator$$anonfun$next$1.apply(Stream.sca=
la:980)
>>>       at
>>>=20
> =
scala.collection.immutable.StreamIterator$$anonfun$next$1.apply(Stream.sca=
la:980)
>>>       at
>>>=20
> =
scala.collection.immutable.StreamIterator$LazyCell.v$lzycompute(Stream.sca=
la:969)
>>>       at
>>> =
scala.collection.immutable.StreamIterator$LazyCell.v(Stream.scala:969)
>>>       at
>>> scala.collection.immutable.StreamIterator.hasNext(Stream.scala:974)
>>>       at =
scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>       at
>>>=20
> =
org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalA=
ppendOnlyMap.scala:137)
>>>       at
>>> org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:58)
>>>       at
>>>=20
> =
org.apache.spark.shuffle.hash.HashShuffleWriter.write(HashShuffleWriter.sc=
ala:55)
>>>       at
>>>=20
> =
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)=

>>>       at
>>>=20
> =
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)=

>>>       at org.apache.spark.scheduler.Task.run(Task.scala:54)
>>>       at
>>> =
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
>>>       at
>>>=20
> =
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.=
java:895)
>>>       at
>>>=20
> =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:918)
>>>       at java.lang.Thread.run(Thread.java:695)
>>>=20
>>>=20
>>=20
>=20
>=20