beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniruddh Sharma <>
Subject Re: File IO - Customer Encrypted keys
Date Tue, 07 Aug 2018 20:05:54 GMT
Thanks Lukasz for revert.
a) Have a further question. I may not be understanding it right. If I embed
this logic in ParDo then won't logic be in a way that it limits file read
to only one file as directly writing in ParDo  or will it still be a
parallel operation .If it is parallel operation then please elaborate more.

b) Is it possible to extend TextIO or AvroIO to decrypt keys (customer
supplied) , if yes then request to elaborate how to do it.



On Mon, Aug 6, 2018 at 8:30 PM, Lukasz Cwik <> wrote:

> I'm assuming your talking about using the Java SDK. Additional details
> about which SDK language, filesystem, and KMS you want to access would be
> useful if the approach below doesn't work for you.
> Your best bet would be use FileIO[1] followed by a ParDo that accesses the
> KMS to get the decryption key and wraps the readable channel with a
> decryptor.
> Note that you'll need to apply your own file parsing logic to pull out the
> records as you won't be able to use things like AvroIO/TextIO to do the
> record parsing for you.
> 1:
> 5.0/org/apache/beam/sdk/io/FileIO.html
> On Tue, Jul 31, 2018 at 10:27 AM <
>> wrote:
>> what is the best suggested way to to read a file encrypted with a
>> customer key which is also wrapped in KMS

View raw message