flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Jain <aj201...@gmail.com>
Subject Re: Reading Data from zip/gzip
Date Mon, 22 Oct 2018 09:16:30 GMT
Hi Chris,

FileInputFormat automatically takes cares of file decompression for the
files with gzip, xz, bz2 and deflate extensions.

--
Thanks,
Amit

Source:
https://github.com/apache/flink/blob/7b040b915504e59243c642b1f4a84c956d96d134/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L118

private static void initDefaultInflaterInputStreamFactories() {
   InflaterInputStreamFactory<?>[] defaultFactories = {
         DeflateInflaterInputStreamFactory.getInstance(),
         GzipInflaterInputStreamFactory.getInstance(),
         Bzip2InputStreamFactory.getInstance(),
         XZInputStreamFactory.getInstance(),
   };
   for (InflaterInputStreamFactory<?> inputStreamFactory : defaultFactories) {
      for (String fileExtension :
inputStreamFactory.getCommonFileExtensions()) {
         registerInflaterInputStreamFactory(fileExtension, inputStreamFactory);
      }
   }
}


On Mon, Oct 22, 2018 at 2:03 PM chrisr123 <chris.ruegger@gmail.com> wrote:

> I'm able to read normal txt or csv files using Flink,
> but what would I need to do in order to read them if they
> are given to me in zip or gzip format? Assuming I do not want
> to have to unzip them.
> Thanks!
>
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

Mime
View raw message