hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Jira)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-16546) make sure staging committers collect DTs for the staging FS
Date Wed, 04 Sep 2019 12:49:00 GMT
Steve Loughran created HADOOP-16546:

             Summary: make sure staging committers collect DTs for the staging FS
                 Key: HADOOP-16546
                 URL: https://issues.apache.org/jira/browse/HADOOP-16546
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.2.0
            Reporter: Steve Loughran

This is not a problem I've seen in the wild, but I've now encountered a problem with hive
doing something like this

we need to (somehow) make sure that the staging committers collect DTs for the staging dir
FS. If this is the default FS or the same as a source or dest FS, this is handled elsewhere,
but otherwise we need to add the staging fs.

I don;t see an easy way to do this, but we could add a new method to PathOutputCommitter to
collect DTs; FileOutputFormat can invoke this alongside its ongoing collection of tokens for
the output FS. Base impl would be a no-op, obviously.

This message was sent by Atlassian Jira

To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

View raw message