From user-return-35863-archive-asf-public=cust-asf.ponee.io@flink.apache.org Wed Jun 17 14:15:40 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 75911180608 for ; Wed, 17 Jun 2020 16:15:40 +0200 (CEST) Received: (qmail 84444 invoked by uid 500); 17 Jun 2020 14:15:38 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 84426 invoked by uid 99); 17 Jun 2020 14:15:38 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2020 14:15:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D2E0FC007D for ; Wed, 17 Jun 2020 14:15:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.002 X-Spam-Level: X-Spam-Status: No, score=0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=kineteque-com.20150623.gappssmtp.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id aYUEuM4MfnFA for ; Wed, 17 Jun 2020 14:15:36 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::1030; helo=mail-pj1-x1030.google.com; envelope-from=mvillalobos@kineteque.com; receiver= Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 577047DDFA for ; Wed, 17 Jun 2020 14:15:35 +0000 (UTC) Received: by mail-pj1-x1030.google.com with SMTP id h22so1130467pjf.1 for ; Wed, 17 Jun 2020 07:15:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kineteque-com.20150623.gappssmtp.com; s=20150623; h=from:content-transfer-encoding:mime-version:subject:date:references :to:in-reply-to:message-id; bh=40RI0/eT06pC4BqWqMLbBllwSJC/+J6HxQafioa8oX8=; b=ClqMNR1uSLKlyQmHfeE9BxRJ33laf7nKunacI4icl0FnEjtmRvYeMizmGazRAd6jIe Gdys71YJ4c3J0FMdOWIYUZFP/JKldipnJjbWyRaEWH/vg2GtQV6I6KJDgbC3qD1oPkGf JxRHQI5hDD7lZBgJds+WdQK0kY0iekVgGdXopEYWDIpu5apqzPrWURmmF/LzfclDVeGF 6sPtEPEzgueZQqVQa9IWd8rlSO8H8j1DgLFjnDWTnHnGc1nOFBblycWF5GlaZ/fEgyGC aNsHLWwUZwwkOky8UNWwzSDGEsFtJpC/ipRpUvDukeBHDz//0wWhNSed/9aL6/zz8u2U iheA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=40RI0/eT06pC4BqWqMLbBllwSJC/+J6HxQafioa8oX8=; b=rCosYY9xgj/agduNd/mLCy7CJc4Gz1fNdj3P1j6GjrBn5qp7JMnyj4zWubWTUozFDn mLUSCk5D8gSVJkmJ03w6tXDIYs9VQ+ZtPSZtZYvWrqNx/yVyaUSyomlZnjz8ihKcXPOi P5XlA/rj06y6zPOscWvuxiZnc8NbfLTRj3kaejzO9NsK1dwrYo+QGvAYn6rmITlwHN5M Px6SenmY64WzSQKqNSp/ioerGyJKU5ceQvHBb3MCGmNwwLvJvUo9hq98mHw36xghKIJt vleqOIs+gODp9hBgtqdGgOmgLAC0+nf+ZIVNfcKI37gssWWLPvVSu/orqIYYL36FH3cy 973g== X-Gm-Message-State: AOAM5327yv3oqXnOU5RSnLFL3KPXueYs7S4DL21gx1Ij5Mxm0gn4ipke ZLWq0qzqOoQeVIiC5KkUEueUd1tOZTs= X-Google-Smtp-Source: ABdhPJzoMB56+NOZa2nDX21LZBzCP1ELEu+HbD93lOPK2sG5SgJfhfV0DB4ld8XBjDBWLLbYU0qeHg== X-Received: by 2002:a17:90a:660b:: with SMTP id l11mr7859075pjj.188.1592403327262; Wed, 17 Jun 2020 07:15:27 -0700 (PDT) Received: from ?IPv6:2605:e000:1319:44b2:7819:bf6b:6133:a6c6? ([2605:e000:1319:44b2:7819:bf6b:6133:a6c6]) by smtp.gmail.com with ESMTPSA id fy21sm411789pjb.38.2020.06.17.07.15.26 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Jun 2020 07:15:26 -0700 (PDT) From: Marco Villalobos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: Does Flink support reading files or CSV files from java.io.InputStream instead of file paths? Date: Wed, 17 Jun 2020 07:15:25 -0700 References: <4E604CF5-FD4B-4DF2-96E2-550AB094813D@kineteque.com> <87512de1-161a-be09-d205-656d8df0e500@apache.org> <3EF3461E-045D-44D1-8EAC-BB901C5935A2@kineteque.com> <1fa42bf1-100b-e8ff-98c0-3902bc39dfa0@apache.org> To: user@flink.apache.org In-Reply-To: <1fa42bf1-100b-e8ff-98c0-3902bc39dfa0@apache.org> Message-Id: <3FD3E310-238A-48AD-9F14-750ABCA4FEC6@kineteque.com> X-Mailer: Apple Mail (2.3608.80.23.2.2) While I still think it would be great for Flink to accept an = InputStream, and allow the programmer to decide if it is a remote TCP = call or local file, for the sake of my demo, I simply=20 found the file path within Gradle and supplied to the Gradle application = run plugin like this: run { args =3D ["--input-file", file('timeseries.csv')] } and that launched my application with minimal configuration. > On Jun 17, 2020, at 7:11 AM, Aljoscha Krettek = wrote: >=20 > Hi, >=20 > for simple demos you can also use env.fromElements() or = env.fromCollection() to create a source from some data that you have = already available. >=20 > Does that help? >=20 > Best, > Aljoscha >=20 > On 16.06.20 15:35, Marco Villalobos wrote: >> Okay, it is not supported. >> I understand such a feature is not needed in production systems, but = it could make testing and demos more portable. I was writing a demo, and = I wanted it to run without command-line arguments, which would have been = very handy. I want the user to simply checkout the code and run it = without having to supply a command line parameter declaring where the = input file resides. >> Thank you. >>> On Jun 16, 2020, at 4:57 AM, Aljoscha Krettek = wrote: >>>=20 >>> Hi Marco, >>>=20 >>> this is not possible since Flink is designed mostly to read files = from a distributed filesystem, where paths are used to refer to those = files. If you read from files on the classpath you could just use plain = old Java code and won't need a distributed processing system such as = Flink. >>>=20 >>> Best, >>> Aljoscha >>>=20 >>> On 16.06.20 06:46, Marco Villalobos wrote: >>>> Does Flink support reading files or CSV files from = java.io.InputStream instead of file paths? >>>> I'd rather just store my file on the class path and load it with = java.lang.ClassLoader#getResourceAsStream(String). >>>> If there is a way, I'd appreciate an example. >>>=20 >=20