Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5531B19651 for ; Thu, 21 Apr 2016 13:17:19 +0000 (UTC) Received: (qmail 99445 invoked by uid 500); 21 Apr 2016 13:17:18 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 99357 invoked by uid 500); 21 Apr 2016 13:17:18 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 99347 invoked by uid 99); 21 Apr 2016 13:17:18 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Apr 2016 13:17:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 2C2C71A06B1 for ; Thu, 21 Apr 2016 13:17:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.898 X-Spam-Level: *** X-Spam-Status: No, score=3.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, MANY_SPAN_IN_TEXT=2.6, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=sics-se.20150623.gappssmtp.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id ARTkYcW5OU7h for ; Thu, 21 Apr 2016 13:17:15 +0000 (UTC) Received: from mail-io0-f173.google.com (mail-io0-f173.google.com [209.85.223.173]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id DC1045FB9D for ; Thu, 21 Apr 2016 13:17:14 +0000 (UTC) Received: by mail-io0-f173.google.com with SMTP id u185so85346710iod.3 for ; Thu, 21 Apr 2016 06:17:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sics-se.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=BHXKkks8l31dIACDiK6MDvYMrN7ArB+Fkpg5CGLE7B4=; b=Ns7t7jKQ8CZbIs05ndX0AmX/B5VGh89POIj5tFVtY4RMOwyy1/vtqpRUSsBaLzDpTP 2Fexq6ND4gw372uXkeb8FgVJV2sm/PneGfXfm2GlIXUZWpyqdVWWWkinVTeqwu1qbwRW bpLryK5rdNxMiAn16LXQel5k+PgQGyuZi1P1f/VR1kKXTJabdSm7stXPmJdRTYRWKyNe rJWubai7sTCcFK8+HqIlITXn+8QoyWmfpmM8bwFHxfqLvb5DXNngq4/YERWpz2LWr8pm gJ2J+FM6U45/b1LEeTEw8g9M8LwNq7sUlriRXbV7KQdfXyy6WrXtEkFl5EB+bxNiP/FA 6wEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=BHXKkks8l31dIACDiK6MDvYMrN7ArB+Fkpg5CGLE7B4=; b=EbU/g03Ysl99/avySRGvTnFLVP/p4KOPG0HhMO168KDZTS/WogwNnHLyD0Xshg08Mt nA/fKz+H7DPAPbNOTXiPcXK01nn8IGXnNRGIWZb3vKHO1HD0wVDpl1PvHaaP4oytpTw3 wnIOMWGVYIlpOarslfjES0Rv7qJuaYVR2/CB1xNsOWF1uCsig6Fu4q+DuPENolsGtnVz bmkxoHRtSrmUUoWojEyWoclEUxIdEUKyXs1NOvMrW0hIobstkUPdYIe2QhUugJplHtsF /eSul6pY+LU9QvOcks9DncEgEiuLmK6fQl4pfUzRNeEpUUJrRKqCgHpuJB2nS0OzaD7G 55jQ== X-Gm-Message-State: AOPr4FXdtpsVK0/PmZLvGEQ3aBTp/mI13e+ytRwjpbjN8VcbZgOiSq806h1T8xaAXOV1OwHPoRMIqxASlPEqlfdx MIME-Version: 1.0 X-Received: by 10.107.171.198 with SMTP id u189mr12801590ioe.111.1461244633805; Thu, 21 Apr 2016 06:17:13 -0700 (PDT) Received: by 10.107.2.76 with HTTP; Thu, 21 Apr 2016 06:17:13 -0700 (PDT) In-Reply-To: References: Date: Thu, 21 Apr 2016 15:17:13 +0200 Message-ID: Subject: Re: Help with generics From: Martin Neumann To: user@flink.apache.org Content-Type: multipart/alternative; boundary=94eb2c05a384541d630530fe870f --94eb2c05a384541d630530fe870f Content-Type: text/plain; charset=UTF-8 Hej, I already tried TypeInformation.of(initModel.class) and it complained that initModel class is unknown. (Since it's of type M) I added a function to the model.class that returns the TypeInformation its working now though I still don't understand what happend behind the scenes and what I changed :-) cheers Martin On Thu, Apr 21, 2016 at 2:27 PM, Aljoscha Krettek wrote: > Hi, > you're right there is not much (very little) in the documentation about > TypeInformation. There is only the description in the JavaDoc: > TypeInformation > Essentially, > how it works is that we always use a TypeSerializer when we have to > serialize values (for sending across network, storing in state, etc.). A > TypeSerializer is created from a TypeInformation and TypeInformation > can be obtained in several ways: > > - the TypeExtractor tries to analyze user functions to determine a > TypeInformation for the input and output type > - the TypeExtractor can try and analyze a given Class to determine a > TypeInformation > - the Scala API uses macros and implicit parameters to create > TypeInformation > - TypeHint can be created to retrieve a TypeInformation > - a TypeInformation can be manually constructed > > tl;dr In your case you can try TypeInformation.of(initModel.class). If > that doesn't work you can try and pass in a function that gives you a > TypeInformation for your model type M. > > Cheers, > Aljoscha > > On Thu, 21 Apr 2016 at 14:16 Martin Neumann wrote: > >> Hej, >> >> I pass an instance of M in the constructor of the class, can I use that >> instead? Maybe give the class a function that returns the right >> TypeInformation? I'm trying figure out how TypeInformation works to better >> understand the Issue is there any documentation about this? At the moment I >> don't really understand what TypeInformation does and how it works. >> >> cheers Martin >> >> On Thu, Apr 21, 2016 at 2:08 PM, Aljoscha Krettek >> wrote: >> >>> Hi, >>> I think it doesn't work because the concrete type of M is not available >>> to create a TypeInformation for M. What you can do is manually pass a >>> TypeInformation or a TypeSerializer to the AnomalyFlatMap and use >>> that when creating the state descriptor. >>> >>> Cheers, >>> Aljoscha >>> >>> On Thu, 21 Apr 2016 at 13:45 Martin Neumann wrote: >>> >>>> Hey, >>>> >>>> I have a FlatMap that uses some generics (appended at the end of the >>>> mail). >>>> I have some trouble with the type inference running into >>>> InvalidTypesException on the first line in the open function. >>>> >>>> How can I fix it? >>>> >>>> Cheers Martin >>>> >>>> >>>> >>>> >>>> public class AnomalyFlatMap extends RichFlatMapFunction, Tuple2> { >>>> private transient ValueState microModel; >>>> private final double threshold; >>>> private boolean updateIfAnomaly; >>>> private M initModel; >>>> >>>> public AnomalyFlatMap(double threshold, M model, boolean updateIfAnomaly) { >>>> this.threshold = threshold; >>>> this.updateIfAnomaly = updateIfAnomaly; >>>> this.initModel = model; >>>> >>>> } >>>> >>>> @Override >>>> public void open(Configuration parameters) throws Exception { >>>> ValueStateDescriptor descriptor = >>>> new ValueStateDescriptor<>( >>>> "RollingMicroModel", >>>> TypeInformation.of(new TypeHint() { >>>> }),initModel >>>> ); >>>> microModel = getRuntimeContext().getState(descriptor); >>>> } >>>> >>>> @Override >>>> public void flatMap(Tuple2 sample, Collector> collector) throws Exception { >>>> M model = microModel.value(); >>>> Anomaly res = model.calculateAnomaly(sample.f0); >>>> >>>> if ( res.getScore() <= threshold || updateIfAnomaly){ >>>> model.addWindow(sample.f0); >>>> microModel.update(model); >>>> } >>>> collector.collect(new Tuple2<>(res,sample.f1)); >>>> } >>>> } >>>> >>>> >>>> >> --94eb2c05a384541d630530fe870f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hej,

I already tried TypeInformation.of(initModel.class) and it complained that initModel class is unknown. (Since i= t's of type M)
I added a function to the model.class that returns the TypeInformation its= working now though I still don't understand what happend behind the sc= enes and what I changed :-)

cheers Martin


On Thu, Apr 21, 2016 at 2:27 PM, A= ljoscha Krettek <aljoscha@apache.org> wrote:
Hi,
you're right there is not= much (very little) in the documentation about TypeInformation. There is on= ly the description in the JavaDoc:=C2=A0TypeInformation=C2=A0Essentia= lly, how it works is that we always use a TypeSerializer<T> when we h= ave to serialize values (for sending across network, storing in state, etc.= ). A TypeSerializer<T> is created from a TypeInformation<T> and= TypeInformation can be obtained in several ways:

= =C2=A0- the TypeExtractor tries to analyze user functions to determine a Ty= peInformation for the input and output type
=C2=A0- the TypeExtra= ctor can try and analyze a given Class<T> to determine a TypeInformat= ion
=C2=A0- the Scala API uses macros and implicit parameters to = create TypeInformation
=C2=A0- TypeHint can be created to retriev= e a TypeInformation
=C2=A0- a TypeInformation can be manually con= structed

tl;dr In your case you can try TypeInform= ation.of(initModel.class). If that doesn't work you can try and pass in= a function that gives you a TypeInformation for your model type M.

Cheers,
Aljoscha

On Thu= , 21 Apr 2016 at 14:16 Martin Neumann <mneumann@sics.se> wrote:
Hej,

I pass an instan= ce of M in the constructor of the class, can I use that instead? Maybe give= the class a function that returns the right TypeInformation? I'm tryin= g figure out how TypeInformation works to better understand the Issue is th= ere any documentation about this? At the moment I don't really understa= nd what TypeInformation does and how it works.=C2=A0

cheers Martin

On Thu, Apr 21, 2016 at 2:08 PM, Aljoscha Krettek <aljoscha@a= pache.org> wrote:
Hi,
I think it doesn't work because the concrete type of M= is not available to create a TypeInformation for M. What you can do is man= ually pass a TypeInformation<M> or a TypeSerializer<M> to the A= nomalyFlatMap and use that when creating the state descriptor.
Cheers,
Aljoscha

On Thu, 21 Apr 2016 at 13:45 Martin Neuma= nn <mneumann@sics.= se> wrote:
= Hey,

I have a FlatMap that uses some generics (appended = at the end of the mail).
I have some trouble with the type infere= nce running into InvalidTypesException on the first line in the open functi= on.

How can I fix it?=C2=A0

Cheers Martin



public class AnomalyFlatMap<M e=
xtends Model,V =
extends ModelValue, T> extends RichFlatMapFunction<Tuple2<V, T>, Tuple2<Anomaly,T>> {
   =
 private transient ValueState&=
lt;M> microModel;<=
br>    private final double threshold;
  =
  private boolean updateIfAno=
maly;
    private M initModel;

    public =
AnomalyFlatMap(double threshol=
d, M model, boolean updateIfAnomaly) {
        th=
is.threshold =3D thres=
hold;
        this.updateIfAnomaly =3D updateIfAnomaly;
        this=
.initModel =3D model;

    }

  =
  @Override
    pub=
lic void open(Configur=
ation parameters) throws Excep=
tion {
        ValueStateDescriptor<M> descriptor =3D
                new ValueStateDescriptor<>(
                =
        "RollingMicroModel"=
,
                        TypeInformation.of(n=
ew TypeHint<M>() =
{
                        }),initModel
                        );
     =
   microModel =3D getR=
untimeContext().getState(descriptor);=

    }

    @Override
    public =
void flatMap(Tuple2<=
;V, T> sampl=
e, Collector<Tuple2<Anom=
aly, T>> collector) throws Exception {
        M model =3D microModel.value();
        Anomaly res  =3D model.calculateA=
nomaly(sample.f0);

        if ( res.=
getScore() <=3D threshold =
|| updateIfAnomaly){
     =
       model.addWindow(sample.f0);
            =
microModel.update(model);
<=
/span>        }
        col=
lector.collect(new Tuple2<&=
gt;(res,sample.f1));<=
br>    }
}



--94eb2c05a384541d630530fe870f--