flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Groesbeck <peter.groesb...@gmail.com>
Subject Does Flink use EMRFS?
Date Fri, 22 May 2020 20:39:34 GMT
Hi,

I'm using Flink StreamingFileSink running in one AWS account (A) to another
(B). I'm also leveraging a SecurityConfiguration in the CFN to assume a
role in account B so that when I write there the files are owned by account
B which then in turn allows account B to delegate to other AWS accounts (C
and D). The reason these files must be owned by the other account is
because AWS doesn't support cross account delegation:
https://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example4.html

SecurityConfiguration:
  Type: AWS::EMR::SecurityConfiguration
  Properties:
    Name: String
    SecurityConfiguration:
      AuthorizationConfiguration:
        EmrFsConfiguration:
          RoleMappings:
            - Role: arn:aws:iam::<B-account>:role/EMR_EC2_DefaultRole
              IdentifierType: Prefix
              Identifiers:
                - s3://my-bucket/prefix/
            - Role: arn:aws:iam::<B-account>:role/EMR_DefaultRole
              IdentifierType: Prefix
              Identifiers:
                - s3://my-bucket/prefix/


I've referenced this in my Cluster block as well:

ReleaseLabel: !Ref ReleaseLabel
SecurityConfiguration: !Ref SecurityConfiguration
ScaleDownBehavior: TERMINATE_AT_TASK_COMPLETION

For some reason the files are still owned by account A. It seems like Flink
is using the old Hadoop FS implementation instead of EMRFS which should (I
believe) grant the proper ownership so that bucket permissions can apply to
the written objects and in turn delegate read permissinos to accounts C, D
ect.

Any help would be greatly appreciated.

Thanks,
Peter

Mime
View raw message