mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: [DISCUSS] More meaningful error when running on Spark 2.0
Date Tue, 15 Nov 2016 17:23:10 GMT
+1 on version checking.
And, there's a little bug as well. this error is technically generated by
something like

dense(Set.empty[Vector]),

i.e., it cannot form a matrix out of an empty collection of vectors. While
this is true, i suppose it needs a `require(...)` insert there to generate
a more meaningful response instead of allowing Scala complaining about
empty collection.

-d


On Mon, Nov 14, 2016 at 7:32 AM, Andrew Palumbo <ap.dev@outlook.com> wrote:

> +1
>
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
> -------- Original message --------
> From: Trevor Grant <trevor.d.grant@gmail.com>
> Date: 11/14/2016 6:49 AM (GMT-08:00)
> To: dev@mahout.apache.org
> Subject: [DISCUSS] More meaningful error when running on Spark 2.0
>
> Hi,
>
> currently when running on Spark 2.0 the user will hit some sort of error,
> one such error is:
>
> java.util.NoSuchElementException: next on empty iterator
> at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
> at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
> at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)
> at scala.collection.IterableLike$class.head(IterableLike.scala:107)
> at scala.collection.mutable.ArrayOps$ofRef.scala$collection$Ind
> exedSeqOptimized$$super$head(ArrayOps.scala:186)
> at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOp
> timized.scala:126)
> at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:186)
> at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl
> y(package.scala:155)
> at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl
> y(package.scala:133)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver
> sableLike.scala:234)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver
> sableLike.scala:234)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSe
> qOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at org.apache.mahout.math.scalabindings.package$.dense(package.scala:133)
> at org.apache.mahout.sparkbindings.SparkEngine$.drmSampleKRows(
> SparkEngine.scala:289)
> at org.apache.mahout.math.drm.package$.drmSampleKRows(package.scala:149)
> at org.apache.mahout.math.drm.package$.drmSampleToTSV(package.scala:165)
> ... 58 elided
>
> With the recent Zeppelin-Mahout integration, there are going to be a lot of
> users unknowingly attempting to run on Mahout on Spark 2.0.  I think it
> would be simple to implement yet save a lot of time on the Zeppelin and
> Mahout mailing lists to do something like:
>
> if sc.version > 1.6.2 then:
>    error("Spark versions ${sc.verion} isn't supported.  Please see
> MAHOUT-... (appropriate jira info)")
>
> I'd like to put something together and, depending on how many issues people
> have on Zeppelin list, be prepared to do a hotfix on 0.12.2 if it becomes
> prudent.  Everyone always complaining that Zeppelin doesn't work because of
> some mystical error, is bad pr.  It DOES say in the notebook and elsewhere
> that we're not 2.0 compliant, however one of the advantages/drawbacks of
> Zeppelin is that without having to really know what you're doing you can
> get a functional local cluster of Flink, Spark, etc. all going.
>
> So we easily could have a space where someone read none of the docs, and is
> whining.  Surely few if any would ever do such a thing, but still I think a
> prudent fix to have in the back pocket.
>
> tg
>
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
>
> *"Fortunate is he, who is able to know the causes of things."  -Virgil*
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message