Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3ADD617E95 for ; Tue, 10 Mar 2015 23:19:35 +0000 (UTC) Received: (qmail 86715 invoked by uid 500); 10 Mar 2015 23:19:35 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 86646 invoked by uid 500); 10 Mar 2015 23:19:35 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 86637 invoked by uid 99); 10 Mar 2015 23:19:35 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Mar 2015 23:19:35 +0000 Received: from mail-qc0-f177.google.com (mail-qc0-f177.google.com [209.85.216.177]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 83E741A02C0 for ; Tue, 10 Mar 2015 23:19:34 +0000 (UTC) Received: by qcvx3 with SMTP id x3so6218731qcv.0 for ; Tue, 10 Mar 2015 16:19:33 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.140.22.234 with SMTP id 97mr42977870qgn.52.1426029573440; Tue, 10 Mar 2015 16:19:33 -0700 (PDT) Received: by 10.229.123.66 with HTTP; Tue, 10 Mar 2015 16:19:33 -0700 (PDT) In-Reply-To: <1426003211656-820.post@n4.nabble.com> References: <1426003211656-820.post@n4.nabble.com> Date: Wed, 11 Mar 2015 00:19:33 +0100 Message-ID: Subject: Re: IDE complains my Filter operator in Scala From: Till Rohrmann To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a11c14f262a0fcf0510f76210 --001a11c14f262a0fcf0510f76210 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Hung, the problem is that the data set pIdIndex contains elements of type Tuple2[String, Int] whereas the defined filter function expects AnnotatedVertex elements. Removing the trailing map operation where you read the vertices should solve this problem. Another problem is that in the open method of the RichFilterFunction you call map on a Java list. The method getBroadcastVariable returns a java.util.List object which does not support this method. You can easily solve this problem by importing scala.collection.JavaConverters.asScalaBufferConverter in the open method scope and then call getRuntimeContext.getBroadcastVariable[Tuple1[Int]]("vertices").asScala.map { _._1 }.toSet If you should encounter other problems, then let us know. Greets, Till On Tue, Mar 10, 2015 at 5:00 PM, HungChang wrote: > Hi, > > My Filter operator in Scala encounters that IDE complains about the data > type: > "overloaded method value filter with alternatives: (fun: ((String, Int)) = =E2=87=92 > Boolean cannot be applied..." > > Scala for me is quite new. I'm thinking the problem comes from the type > doesn't match each other in map operator and filter operator: > > val pldIndex =3D GraphUtils.readVertices(PLDIndexFile).map { vertex =3D> > Tuple2(vertex.annotation, vertex.id) } > > class PruneToSampleVerticesFilter extends > RichFilterFunction[AnnotatedVertex] > > Did you have any suggestion that in which direction should I work on to > solve this? > > Best regards, > > Hung > > > -------------------------------------------------------------------------= --------------------------------- > def compute(trackingGraphFile: String, PLDIndexFile: String, outputPath: > String) =3D { > > implicit val env =3D ExecutionEnvironment.getExecutionEnvironment > > // Vertex =3D> VertexName, VertexID > val pldIndex =3D GraphUtils.readVertices(PLDIndexFile).map { vertex = =3D> > Tuple2(vertex.annotation, vertex.id) } > val trackingGraphEdges =3D GraphUtils.readEdges(trackingGraphFile) > > val trackingGraphVertices =3D trackingGraphEdges.map { edge =3D> > Tuple1(edge.target) } > .distinct > > // IDE complains this filter > val prunedPldIndex =3D pldIndex.filter(new PruneToSampleVerticesFilte= r) > > .withBroadcastSet(trackingGraphVertices, "vertices") > > prunedPldIndex.writeAsCsv(outputPath, fieldDelimiter =3D "\t", writeM= ode > =3D > WriteMode.OVERWRITE) > > env.execute() > } > > class PruneToSampleVerticesFilter extends > RichFilterFunction[AnnotatedVertex] { > > var vertices: Set[Int] =3D null > > override def open(parameters: Configuration) =3D { > vertices =3D > getRuntimeContext.getBroadcastVariable[Tuple1[Int]]("vertices").map { _._= 1 > } > > .toSet > } > > override def filter(vertex: AnnotatedVertex): Boolean =3D { > vertices.contains(vertex.id) > } > } > > > -------------------------------------------------------------------------= ----------------------------------------- > // case class > case class AnnotatedVertex(annotation: String, id: Int) > case class Edge(src: Int, target: Int) > > > > -- > View this message in context: > http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble= .com/IDE-complains-my-Filter-operator-in-Scala-tp820.html > Sent from the Apache Flink (Incubator) User Mailing List archive. mailing > list archive at Nabble.com. > --001a11c14f262a0fcf0510f76210 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Hung,

the problem is that the data s= et pIdIndex contains elements of type Tuple2[String, Int] whereas the defin= ed filter function expects AnnotatedVertex elements. Removing the trailing = map operation where you read the vertices should solve this problem.
<= div>
Another problem is that in the open method of the RichFi= lterFunction you call map on a Java list. The method getBroadcastVariable r= eturns a java.util.List object which does not support this method. You can = easily solve this problem by importing scala.collection.JavaConverters.asSc= alaBufferConverter in the open method scope and then call=C2=A0getRuntimeContext.getBroadcastVariable[Tuple1[Int]]("vertices").asScala.map {= _._1 }.toSet

If you = should encounter other problems, then let us know.

Greets,

Till

On Tue, Mar 10, 2015 at 5:00 PM, HungChang <unicorn.banachi@gmail.com> wrote:
Hi,

My Filter operator in Scala encounters that IDE complains about the data type:
"overloaded method value filter with alternatives: (fun: ((String, Int= )) =E2=87=92
Boolean cannot be applied..."

Scala for me is quite new. I'm thinking the problem comes from the type=
doesn't match each other in map operator and filter operator:

val pldIndex =3D GraphUtils.readVertices(PLDIndexFile).map { vertex =3D>=
Tuple2(vertex.annotation, ve= rtex.id) }

class PruneToSampleVerticesFilter extends
RichFilterFunction[AnnotatedVertex]

Did you have any suggestion that in which direction should I work on to
solve this?

Best regards,

Hung

---------------------------------------------------------------------------= -------------------------------
def compute(trackingGraphFile: String, PLDIndexFile: String, outputPath: String) =3D {

=C2=A0 =C2=A0 implicit val env =3D ExecutionEnvironment.getExecutionEnviron= ment

=C2=A0 =C2=A0 // Vertex =3D> VertexName, VertexID
=C2=A0 =C2=A0 val pldIndex =3D GraphUtils.readVertices(PLDIndexFile).map { = vertex =3D>
Tuple2(vertex.annotation, ve= rtex.id) }
=C2=A0 =C2=A0 val trackingGraphEdges =3D GraphUtils.readEdges(trackingGraph= File)

=C2=A0 =C2=A0 val trackingGraphVertices =3D trackingGraphEdges.map { edge = =3D>
Tuple1(edge.target) }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 .distinct

// IDE complains this filter
=C2=A0 =C2=A0 val prunedPldIndex =3D pldIndex.filter(new PruneToSampleVerti= cesFilter)

.withBroadcastSet(trackingGraphVertices, "vertices")

=C2=A0 =C2=A0 prunedPldIndex.writeAsCsv(outputPath, fieldDelimiter =3D &quo= t;\t", writeMode =3D
WriteMode.OVERWRITE)

=C2=A0 =C2=A0 env.execute()
=C2=A0 }

=C2=A0 class PruneToSampleVerticesFilter extends
RichFilterFunction[AnnotatedVertex] {

=C2=A0 =C2=A0 var vertices: Set[Int] =3D null

=C2=A0 =C2=A0 override def open(parameters: Configuration) =3D {
=C2=A0 =C2=A0 =C2=A0 vertices =3D
getRuntimeContext.getBroadcastVariable[Tuple1[Int]]("vertices").m= ap { _._1 }

.toSet
=C2=A0 =C2=A0 }

=C2=A0 =C2=A0 override def filter(vertex: AnnotatedVertex): Boolean =3D { =C2=A0 =C2=A0 =C2=A0 vertices.contains(vertex.id)
=C2=A0 =C2=A0 }
=C2=A0 }

---------------------------------------------------------------------------= ---------------------------------------
// case class
case class AnnotatedVertex(annotation: String, id: Int)
case class Edge(src: Int, target: Int)



--
View this message in context: http://apache-flink-incubator-user= -mailing-list-archive.2336050.n4.nabble.com/IDE-complains-my-Filter-operato= r-in-Scala-tp820.html
Sent from the Apache Flink (Incubator) User Mailing List archive. mailing l= ist archive at Nabble.com.

--001a11c14f262a0fcf0510f76210--