Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6341D18AD1 for ; Thu, 4 Feb 2016 14:14:27 +0000 (UTC) Received: (qmail 82026 invoked by uid 500); 4 Feb 2016 14:14:05 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 81934 invoked by uid 500); 4 Feb 2016 14:14:05 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 81924 invoked by uid 99); 4 Feb 2016 14:14:05 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2016 14:14:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B51B0C0E4F for ; Thu, 4 Feb 2016 14:14:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=googlemail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id wnkSW40iPbrv for ; Thu, 4 Feb 2016 14:14:02 +0000 (UTC) Received: from mail-wm0-f43.google.com (mail-wm0-f43.google.com [74.125.82.43]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 18F4820546 for ; Thu, 4 Feb 2016 14:14:02 +0000 (UTC) Received: by mail-wm0-f43.google.com with SMTP id 128so28960914wmz.1 for ; Thu, 04 Feb 2016 06:14:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=from:content-type:subject:message-id:date:to:mime-version; bh=sSfmxz/59tQ2iEy4CbedcO0b+vVvTRp62gdIZXjTFDs=; b=S2Tc+KxRoIFhCXFTpGlpASkOTyzakGwWFHfUmPEjywkXkFsTsW6KzKJ8FKrZ1cTmKS Om36OiSBkFMQz1hlCbk9r6+wyslfRBaUksfncyC+GMTAZscK5/3PJw1JN1BGJ4Sffn7f pfVNy9/f7JZKBujNFPIh7bnDy4rAQpAbs3FNacfI/2naou3lGRTcuO9y5Q42S2/oKTca 27m2OnZ026VqzmignTK9p9T1rwRn8FA9fA6xpYHAFVg3XqgdIgCtLZ171FQxafEBZ4RZ BFNEMmg2UE3lEZIXFWnqiGuKEZOXuIHaS6cFgmBJn/y55MJX2vm8QK+yI0hkztKxRWSH dWMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:subject:message-id:date:to :mime-version; bh=sSfmxz/59tQ2iEy4CbedcO0b+vVvTRp62gdIZXjTFDs=; b=CtjqlrgO1V5a07ZxudwJTl0dP/0rPF6gPF/baXtW+PpGnQPHR1Su/94E0sDkx/HjGo E/tlshcu08LU0z36ogyoJ5CGvtv8AuHLmL2WBHpH7iDrQpzuHVpVTgix7hXisTVHSYtd fc14fXqqW+8CJML+h+ugF3lAEMi4gM+t7N8Zng9wBOu5KfXPZOrhDZ5xY9MAyY2PcVel 4KNZUIEAVnn2sbhNDCVFLBAaSnpUANL48E4HG20ymNK11L5KmC9lL0OcxNVGX+CSaKa1 YBu9EimbF/ixrytqYBAJE4nGGpm6GRsh0QLloiqFZ29xCOhgra05eOa6uXIDapQZnk+H 04Pg== X-Gm-Message-State: AG10YOTF5GtAIBgTNs9kQ9uFLKFuahlfDm5IkYfD6Smx9V77Af488mE0HVhMuGdvP1OCnQ== X-Received: by 10.194.9.98 with SMTP id y2mr8606289wja.105.1454595240736; Thu, 04 Feb 2016 06:14:00 -0800 (PST) Received: from [192.168.2.101] (dslb-088-075-246-090.088.075.pools.vodafone-ip.de. [88.75.246.90]) by smtp.googlemail.com with ESMTPSA id x186sm26376607wmg.19.2016.02.04.06.13.59 for (version=TLSv1/SSLv3 cipher=OTHER); Thu, 04 Feb 2016 06:13:59 -0800 (PST) From: Lydia Ickler Content-Type: multipart/alternative; boundary="Apple-Mail=_CAD55D98-8485-41E7-A712-FCD7D21B9100" Subject: DistributedMatrix in Flink Message-Id: Date: Thu, 4 Feb 2016 15:13:58 +0100 To: user@flink.apache.org Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) X-Mailer: Apple Mail (2.3112) --Apple-Mail=_CAD55D98-8485-41E7-A712-FCD7D21B9100 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi all, as mentioned before I am trying to import the RowMatrix from Spark to = Flink=E2=80=A6 In the code I already ran into a dead end=E2=80=A6 In the function = multiplyGramianMatrixBy() (see end of mail) there is the line:=20 rows.context.broadcast(v) (rows is a DataSet[Vector] What exactly is this line doing? Does it fill the =E2=80=9Econtent=E2=80=9C= of v into the variable rows? And another question: What is the function treeAggregate doing ? And how would you tackle a = =E2=80=9Ecopy=E2=80=9C of that in Flink? Thanks in advance! Best regards,=20 Lydia private[flink] def multiplyGramianMatrixBy(v: DenseVector[Double]): = DenseVector[Double] =3D { val n =3D numCols().toInt val vbr =3D rows.context.broadcast(v) rows.treeAggregate(BDV.zeros[Double](n))( seqOp =3D (U, r) =3D> { val rBrz =3D r.toBreeze val a =3D rBrz.dot(vbr.data) rBrz match { // use specialized axpy for better performance case _: BDV[_] =3D> brzAxpy(a, rBrz.asInstanceOf[BDV[Double]], = U) case _: BSV[_] =3D> brzAxpy(a, rBrz.asInstanceOf[BSV[Double]], = U) case _ =3D> throw new UnsupportedOperationException( s"Do not support vector operation from type = ${rBrz.getClass.getName}.") } U }, combOp =3D (U1, U2) =3D> U1 +=3D U2) } --Apple-Mail=_CAD55D98-8485-41E7-A712-FCD7D21B9100 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi all,

as = mentioned before I am trying to import the RowMatrix from Spark to = Flink=E2=80=A6

In the code I already ran into a dead end=E2=80=A6 In the = function multiplyGramianMatrixBy() (see end of mail) there is the = line: 
rows.context.broadcast(v) (rows is a = DataSet[Vector]
What exactly = is this line doing? Does it fill the =E2=80=9Econtent=E2=80=9C of v = into the variable rows?
And another question:
What is the = function treeAggregate doing ? And how would you tackle = a =E2=80=9Ecopy=E2=80=9C of that in Flink?

Thanks in = advance!
Best regards, 
Lydia


private[flink] def =
multiplyGramianMatrixBy(v: DenseVector[Double]): =
DenseVector[Double] =3D {
  val n =3D =
numCols().toInt

  val vbr =3D =
rows.context.broadcast(v)

  =
rows.treeAggregate(BDV.zeros[Double](n))(
    seqOp =3D (U, r) =
=3D> {
      val rBrz =3D =
r.toBreeze
      val a =3D =
rBrz.dot(vbr.data)
      rBrz match {
        // use specialized axpy for better performance
        case _: BDV[_] =3D> brzAxpy(a, =
rBrz.asInstanceOf[BDV[Double]], U)
        case _: =
BSV[_] =3D> brzAxpy(a, rBrz.asInstanceOf[BSV[Double]], U)
        case _ =3D> throw new =
UnsupportedOperationException(
          s"Do not support =
vector operation from type ${rBrz.getClass.getName}.")
      }
      U
    }, combOp =3D =
(U1, U2) =3D> U1 +=3D U2)
}

= --Apple-Mail=_CAD55D98-8485-41E7-A712-FCD7D21B9100--