From user-return-21348-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Jul 17 06:28:50 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 296AB180600 for ; Tue, 17 Jul 2018 06:28:49 +0200 (CEST) Received: (qmail 93144 invoked by uid 500); 17 Jul 2018 04:28:48 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 93134 invoked by uid 99); 17 Jul 2018 04:28:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jul 2018 04:28:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6FC7F1A0EBF for ; Tue, 17 Jul 2018 04:28:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id baJmuQShqR7W for ; Tue, 17 Jul 2018 04:28:46 +0000 (UTC) Received: from mandala.kddilabs.jp (mandala.kddilabs.jp [192.26.91.6]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 11C2B5F27D for ; Tue, 17 Jul 2018 04:28:45 +0000 (UTC) Received: from localhost (mandala.kddilabs.jp [127.0.0.1]) by mandala.kddilabs.jp (Postfix) with ESMTP id 4B422C0EF65E; Tue, 17 Jul 2018 13:28:39 +0900 (JST) X-Virus-Scanned: amavisd-new at kddi-research.jp Received: from mandala.kddilabs.jp ([127.0.0.1]) by localhost (mandala.kddilabs.jp [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5mzEdu76ecYG; Tue, 17 Jul 2018 13:28:37 +0900 (JST) Received: from safeattach.localdomain (unknown [IPv6:2001:200:601:1a00:20c:29ff:fe79:2280]) by mandala.kddilabs.jp (Postfix) with ESMTP id F07C7C0EF659; Tue, 17 Jul 2018 13:27:37 +0900 (JST) Received: from [127.0.0.1] (dhcp132.west-4f.cn.kddilabs.jp [172.19.124.132]) by safeattach.localdomain with ESMTP id w6H4RbbY010111; Tue, 17 Jul 2018 13:27:37 +0900 Subject: Re: Some question about document To: vino yang Cc: user@flink.apache.org References: <98eb9b7f-8eed-2eb2-21e4-fa97377da655@apache.org> <30232e12-4efe-8da3-0617-aa9c184fb92d@kddi-research.jp> From: Yuta Morisawa Message-ID: <9a63c262-8a3d-925d-e4ad-bc174bede567@kddi-research.jp> Date: Tue, 17 Jul 2018 13:27:38 +0900 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Hi yang Thank you for your comment. I read document and got an idea. Flink uses custom serializers on known types, and the fallback is kryo. The other arbitary objects is serialized by kryo. On 2018/07/12 12:14, vino yang wrote: > Hi Yuta, > > It seems Chesnay is right. The "fallback" in flink's documentation is in > terms of the types flink supported. But for all the other arbitrary > types kryo is the first choice. > > 2018-07-12 9:55 GMT+08:00 Yuta Morisawa >: > > Thank you for your answer. > > > For POJOs Flink has a custom serializer. For arbitrary objects we use > > kryo, and can use Avro as a fallback. > https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#serialization-of-pojo-types > > > It may be the reverse. > Kryo is for fallback, right? > > > On 2018/07/11 19:00, Chesnay Schepler wrote: > > 1) TypeInformation are used to create serializers, comparators > and to verify correctness of certain operations (like > projections on tuple datasets). > > 2) see > https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html > > > 3) Flink comes with a number of serializers for varying types as > outlined here > >. > For POJOs Flink has a custom serializer. For arbitrary objects > we use kryo, and can use Avro as a fallback. > > On 11.07.2018 09:24, Yuta Morisawa wrote: > > Hi all > > Now, I'm reading Flink document and I have some points to > feel difficult to get an idea. > I'd appreciate if you tell it me. > > 1,TypeInformation >  I understand TypeInformation is used for selecting > relevant serializer and comparator. >  But, the ducument doesn't specify if it has another way to > be used. > >  So, what I want to know is that what kinds of process gets > benefit from TypeInformation other than serializer and > comparator. > > 2, Managed Memory >  The word "Managed memory" is appeared several time in the > document but I can't find any detail description. >  This is the only document I found > (https://www.slideshare.net/sbaltagi/overview-of-apacheflinkbyslimbaltagi > ) > >  If anyone has document that explains managed memory, > please let me know. > > 3, Serializer >  What do the words in the document  "serializers we ship > with Flink" mean? I know Flink uses avro for POJOs, is it > the same thing? > https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html > > > > > Regards, > Yuta > > > >