Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4FE5A18B07 for ; Tue, 29 Sep 2015 08:00:40 +0000 (UTC) Received: (qmail 78887 invoked by uid 500); 29 Sep 2015 08:00:34 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 78812 invoked by uid 500); 29 Sep 2015 08:00:33 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 78803 invoked by uid 99); 29 Sep 2015 08:00:33 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2015 08:00:33 +0000 Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 547891A0040 for ; Tue, 29 Sep 2015 08:00:33 +0000 (UTC) Received: by laer8 with SMTP id r8so57956267lae.2 for ; Tue, 29 Sep 2015 01:00:31 -0700 (PDT) X-Received: by 10.152.5.133 with SMTP id s5mr6876258las.19.1443513631770; Tue, 29 Sep 2015 01:00:31 -0700 (PDT) MIME-Version: 1.0 References: <1443493777408-2957.post@n4.nabble.com> In-Reply-To: <1443493777408-2957.post@n4.nabble.com> From: Aljoscha Krettek Date: Tue, 29 Sep 2015 08:00:22 +0000 Message-ID: Subject: Re: FLink Streaming - Parallelize ArrayList To: user@flink.apache.org Content-Type: multipart/alternative; boundary=089e013d0ffc40192f0520de3589 --089e013d0ffc40192f0520de3589 Content-Type: text/plain; charset=UTF-8 Hi, I think for what you are trying a Connected FlatMap would be best suited. Using this, you can have an operation that has two input streams: input1 = env.socketStream(...) input2 = env.socketStream(...) result = input1.connect(input2) .flatMap(new CoFlatMapFunction { void flatMap1(...) {...} void flatMap2(...) {...} }) here, the flatMap1 will get elements from the first input stream while flatMap2 will get elements from the second input stream. You can now internally keep a list of elements from the first input. You have to be careful, however, to not let the list get too large. Otherwise, you might blow your JVM with an OutOfMemory exception. Cheers, Aljoscha On Tue, 29 Sep 2015 at 04:38 defstat wrote: > Hi all, > > Is there a way to either Broadcast or Parallelize an ArrayList of custom > objects? > > The application is as follows: > > I have 2 streams taken from two sockets. The first stream contains vectors > that should populate one list of vectors by deciding whether an arriving > vector is part of the list, after calculations made on the existing list > vectors and the new one. > > The other stream is also containing vectors that use the above list, and > sorts it using the info contained in the incoming vectors. > > I tried to solve this using > > private static final ArrayList aSetOfVectors = new > ArrayList(); > > vectorsFirstStream.fold(aSetOfVectors , new FoldFunction ArrayList<Vector>>() { > .... (Calculations and new aSetOfVectors produced) ...... > > return aSetOfVectors; > } > > vectorsSecondStream.fold(aSetOfVectors , new FoldFunction ArrayList<Vector>>() { > .... (Calculations and sorted vector list produced) ...... > } > > I think that that approach could work only locally. Is there a way to > parallelize the above? > > Thank you in advance > > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FLink-Streaming-Parallelize-ArrayList-tp2957.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. > --089e013d0ffc40192f0520de3589 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,
I think for what you are trying a Connected FlatMa= p would be best suited. Using this, you can have an operation that has two = input streams:

input1 =3D env.socketStream(...)
input2 =3D env.socketStream(...)

result = =3D input1.connect(input2)
=C2=A0 =C2=A0 .flatMap(new CoFlatMapFu= nction<IN1, IN2, OUT> {
=C2=A0 =C2=A0 =C2=A0 =C2=A0void fla= tMap1(...) {...}
=C2=A0 =C2=A0 =C2=A0 =C2=A0void flatMap2(...) {.= ..}
=C2=A0 =C2=A0 })

here, the flatMap1 = will get elements from the first input stream while flatMap2 will get eleme= nts from the second input stream. You can now internally keep a list of ele= ments from the first input. You have to be careful, however, to not let the= list get too large. Otherwise, you might blow your JVM with an OutOfMemory= exception.

Cheers,
Aljoscha
=
On Tue, 29 Sep 2015 at 04:3= 8 defstat <defstat@gmail.com>= ; wrote:
Hi all,

Is there a way to either Broadcast or Parallelize=C2=A0 an ArrayList of cus= tom
objects?

The application is as follows:

I have 2 streams taken from two sockets. The first stream contains vectors<= br> that should populate one list of vectors by deciding whether an arriving vector is part of the list, after calculations made on the existing list vectors and the new one.

The other stream is also containing vectors that use the above list, and sorts it using the info contained in the incoming vectors.

I tried to solve this using

private static final ArrayList<Vector> aSetOfVectors =3D new
ArrayList<Vector>();

vectorsFirstStream.fold(aSetOfVectors , new FoldFunction<Vector,
ArrayList&lt;Vector>>() {
=C2=A0.... (Calculations and new aSetOfVectors produced) ......

return aSetOfVectors;
}

vectorsSecondStream.fold(aSetOfVectors , new FoldFunction<Vector,
ArrayList&lt;Vector>>() {
=C2=A0.... (Calculations and sorted vector list produced) ......
}

I think that that approach could work only locally. Is there a way to
parallelize the above?

Thank you in advance




--
View this message in context: http://apache-flink-user-mai= ling-list-archive.2336050.n4.nabble.com/FLink-Streaming-Parallelize-ArrayLi= st-tp2957.html
Sent from the Apache Flink User Mailing List archive. mailing list archive = at Nabble.com.
--089e013d0ffc40192f0520de3589--