Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8F8CF200BA0 for ; Fri, 14 Oct 2016 18:01:05 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8E322160ADD; Fri, 14 Oct 2016 16:01:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id ACCCC160AD3 for ; Fri, 14 Oct 2016 18:01:04 +0200 (CEST) Received: (qmail 65036 invoked by uid 500); 14 Oct 2016 16:01:03 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 65020 invoked by uid 99); 14 Oct 2016 16:01:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Oct 2016 16:01:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 0371B1A04C1 for ; Fri, 14 Oct 2016 16:01:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.679 X-Spam-Level: * X-Spam-Status: No, score=1.679 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id CWxJhDCW-1wU for ; Fri, 14 Oct 2016 16:00:59 +0000 (UTC) Received: from mail-lf0-f48.google.com (mail-lf0-f48.google.com [209.85.215.48]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B7EED5FAC3 for ; Fri, 14 Oct 2016 16:00:58 +0000 (UTC) Received: by mail-lf0-f48.google.com with SMTP id l131so158530829lfl.2 for ; Fri, 14 Oct 2016 09:00:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=d8XI4HE5+bUakxUicvNA1WOMJ3cDaEX8eeynfQm2j9U=; b=zmJwBibb8aKUfIorY4Ity1fS8eP96Os8VoGqCaIWKBiOfTLWJgiiUEjZtV9FitW0qS e1ym4+2V6hfh2xscFHB8z5oePI1Fs2XTtF3/6SbZncj3NzpjS9d5KhK95QsXr4lrFFDI 42FaT8upUP+DCgKqZkEPRgALbN7EEr2KIDqasAlSQlq4OOCmxVnn45x+87xvNmWrDS7/ YATon7o6X9G+J4yOMNwSszxF+NMICgFH1/fMUVcPXqaZysVTHcCcolKZ9YtaZy3dTcld G0tYe2W0q/Twwg9oihM2jy7krjxzF89RTZBW4iM83GWHwBw2Lmrg2MJEI4blPVk4YXUM cybw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=d8XI4HE5+bUakxUicvNA1WOMJ3cDaEX8eeynfQm2j9U=; b=O1ydttUIJzxmJcl28kVOstp5KDaf1LmrbdatwqPrayt1ctS9oRdFUm9e4rOiKo3lyN YX6xbDxYq06XMD0kBzg9rkDokZjvXg7fG2iiiblF6bAIe6WjjBv1l1vXzZKvpagT7luN JybDd3wV8U7zLa+Bw8Ovp1VUSAyMCe8ThWBpJNq+ld92we0bYpEQHKwMpRPvvlWCr055 w2qM8JumvqerXs7lHAkT8AM/+mr9IUDNobKRHlRzfzhRJBxkN8OCkN5Jx6DipO+tufIZ Y7EI1BqYn4vMFNw9hYX9oYgMK9ymERT8j/UCCXnpnJIf2rEF/1zowu/mru1XFZIc5rzE 6F2g== X-Gm-Message-State: AA6/9RkTkUP8bPWQjpH+ai3GxnXYsNANVAs0VHY50HNZDkCVRcj3K4DUgNKRbbztehj2YqvFESA9obZRZ7giog== X-Received: by 10.25.92.203 with SMTP id u72mr4398534lfi.62.1476460854916; Fri, 14 Oct 2016 09:00:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.133.7 with HTTP; Fri, 14 Oct 2016 09:00:24 -0700 (PDT) In-Reply-To: References: From: Fabian Hueske Date: Fri, 14 Oct 2016 18:00:24 +0200 Message-ID: Subject: Re: [DISCUSS] Deprecate Hadoop source method from (batch) ExecutionEnvironment To: Shannon Carey Cc: "user@flink.apache.org" Content-Type: multipart/alternative; boundary=94eb2c0c88c8c8399f053ed554e5 archived-at: Fri, 14 Oct 2016 16:01:05 -0000 --94eb2c0c88c8c8399f053ed554e5 Content-Type: text/plain; charset=UTF-8 Hi Shannon, the plan is as follows: We will keep the methods as they are for 1.2 but deprecate them and at the same time we will add alternatives in an optional dependency. In a later release, the deprecated methods will be removed and everybody has to switch to the optional dependency. Does that work for you? Best, Fabian 2016-10-14 17:30 GMT+02:00 Shannon Carey : > Speaking as a user, if you are suggesting that you will retain the > functionality but move the methods to an optional dependency, it makes > sense to me. We have used the Hadoop integration for > AvroParquetInputFormat and CqlBulkOutputFormat in Flink (although we won't > be using CqlBulkOutputFormat any longer because it doesn't seem to be > reliable). > > -Shannon > > From: Fabian Hueske > Date: Friday, October 14, 2016 at 4:29 AM > To: , "dev@flink.apache.org" > Subject: [DISCUSS] Deprecate Hadoop source method from (batch) > ExecutionEnvironment > > Hi everybody, > > I would like to propose to deprecate the utility methods to read data with > Hadoop InputFormats from the (batch) ExecutionEnvironment. > > The motivation for deprecating these methods is reduce Flink's dependency > on Hadoop but rather have Hadoop as an optional dependency for users that > actually need it (HDFS, MapRed-Compat, ...). Eventually, we want to have > Flink distribution that does not have a hard Hadoop dependency. > > One step for this is to remove the Hadoop dependency from flink-java > (Flink's Java DataSet API) which is currently required due to the above > utility methods (see FLINK-4315). We recently received a PR that addresses > FLINK-4315 and removes the Hadoop methods from the ExecutionEnvironment. > After some discussion, it was decided to defer the PR to Flink 2.0 because > it breaks the API (these methods are delared @PublicEvolving). > > I propose to accept this PR for Flink 1.2, but instead of removing the > methods deprecating them. > This would help to migrate old code and prevent new usage of these methods. > For a later Flink release (1.3 or 2.0) we could remove these methods and > the Hadoop dependency on flink-java. > > What do others think? > > Best, Fabian > --94eb2c0c88c8c8399f053ed554e5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Shannon,

the= plan is as follows:

We will keep the methods as th= ey are for 1.2 but deprecate them and at the same time we will add alternat= ives in an optional dependency.
In a later release, the depr= ecated methods will be removed and everybody has to switch to the optional = dependency.

Does that work for you?

Best, Fabian<= br>

2016-10-= 14 17:30 GMT+02:00 Shannon Carey <scarey@expedia.com>:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
Speaking as a user, if you are suggesting that you will retain the fun= ctionality but move the methods to an optional dependency, it makes sense t= o me. We have used the Hadoop integration for AvroParquetInputFormat=C2=A0a= nd CqlBulkOutputFormat in Flink (although we won't be using CqlBulkOutputFormat any longer because it doesn'= t seem to be reliable).

-Shannon

From: Fabian Hueske <fhueske@gmail.com>
Date: Friday, October 14, 2016 at 4= :29 AM
To: <user@flink.apache.org>, "dev@flink.apache.org" <dev@f= link.apache.org>
Subject: [DISCUSS] Deprecate Hadoop= source method from (batch) ExecutionEnvironment

Hi everybody,

I would like to propose to deprecate the utility methods to read data with = Hadoop InputFormats from the (batch) ExecutionEnvironment.

The motivation for deprecating these methods is reduce Flink's dependen= cy on Hadoop but rather have Hadoop as an optional dependency for users tha= t actually need it (HDFS, MapRed-Compat, ...). Eventually, we want to have = Flink distribution that does not have a hard Hadoop dependency.

One step for this is to remove the Hadoop dependency from flink-java (Flink= 's Java DataSet API) which is currently required due to the above utili= ty methods (see FLINK-4315). We recently received a PR that addresses FLINK= -4315 and removes the Hadoop methods from the ExecutionEnvironment. After some discussion, it was decided to de= fer the PR to Flink 2.0 because it breaks the API (these methods are delare= d @PublicEvolving).

I propose to accept this PR for Flink 1.2, but instead of removing the= methods deprecating them.
This would help to migrate old code and prevent new usage of these met= hods.
For a later Flink release (1.3 or 2.0) we could remove these methods a= nd the Hadoop dependency on flink-java.

What do others think?

Best, Fabian

--94eb2c0c88c8c8399f053ed554e5--