mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Avati <av...@gluster.org>
Subject Re: Problem of dimensions
Date Sun, 20 Jul 2014 03:56:32 GMT
On Sat, Jul 19, 2014 at 6:50 PM, Pat Ferrel <pat.ferrel@gmail.com> wrote:

>
> On another thread I’ll send you code that shows A + 1 works with blank
> rows in A.
>

I don't see how that worked for you. See this:

  test("DRM addToRowCardinality - will fail") {
    val inCoreA = sparse(
      0 -> 1 :: 1 -> 2 :: Nil,
      0 -> 3 :: 1 -> 4 :: Nil,
      0 -> 2 :: 1 -> 0.0 :: Nil
    )

    val inCoreBControl = sparse(
      0 -> 2 :: 1 -> 3 :: Nil,
      0 -> 4 :: 1 -> 5 :: Nil,
      0 -> 3 :: 1 -> 1 :: Nil,
      0 -> 1 :: 1 -> 1 :: Nil,
      0 -> 1 :: 1 -> 1 :: Nil
    )

    val drmA = drmParallelize(inCoreA)
    drmA.addToRowCardinality(2)
    val drmB = (drmA + 1.0).checkpoint()

    (drmB.collect - inCoreBControl).norm should be < 1e-3

  }

  test("DRM addToRowCardinality - wont fail") {
    val inCoreA = sparse(
      0 -> 1 :: 1 -> 2 :: Nil,
      0 -> 3 :: 1 -> 4 :: Nil,
      0 -> 2 :: 1 -> 0.0 :: Nil
    )

    val inCoreBWrong = sparse(
      0 -> 2 :: 1 -> 3 :: Nil,
      0 -> 4 :: 1 -> 5 :: Nil,
      0 -> 3 :: 1 -> 1 :: Nil,
      0 -> 0 :: 1 -> 0 :: Nil,
      0 -> 0 :: 1 -> 0 :: Nil
    )

    val drmA = drmParallelize(inCoreA)
    drmA.addToRowCardinality(2)
    val drmB = (drmA + 1.0).checkpoint()

    (drmB.collect - inCoreBWrong).norm should be < 1e-3
  }


And sure enough, inCoreBControl fails, and inCoreBWrong succeeds:

- DRM addToRowCardinality - will fail *** FAILED ***

  2.0 was not less than 0.001 (DrmLikeSuiteBase.scala:116)
- DRM addToRowCardinality - wont fail


BTW this implies rbind will not solve the problem, it is firmly in data
> prep. But until I know the rules I won’t know how to do the right thing.
>

Rbind expects both A and B to have their Int row keys filled from 0 to
nrow-1, which is how they should be ideally.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message