hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
Date Fri, 17 Nov 2017 20:02:01 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated HADOOP-15003:
    Attachment: HADOOP-13786-051.patch

patch 051

h3. Test failure from reuse of non-magic FS.
* special check in file creation for __magic path and not magic fs, logs @ info. Warn maybe?
* committer troubleshooting  mentions what this warning means
* testing.md lists what we want from good tests (parallelise, etc), and mentions FS caching
* magic tests disable FS Caching.

we may want to simplify things here by always setting the magic build in our test filesystems.
That way: no need to play these games and no recurrent problems. Thoughts?

h3. Isolate S3A filesystem from transitive dependencies on hadoop-mapreduce

Isolate all utility classes which use MR datatypes from the others into their own package,
and the magic path ones, splitting CommitUtils in three. This is needed to ensure s3a loads
when mapreduce JARs aren't on the classpath.

The previous patches probably meant S3AFS  woudn't load without  hadoop-mapreduce on the CP
just because some other static methods in CommitUtils needed it. Partitioning things up and
a full review of the s3 codepath *appears* to fix this. However, I plan to add a new downstream
test module somewhere just to verify classpath setup  by opening the landsat repository and
listing it, 

> Merge S3A committers into trunk: Yetus patch checker
> ----------------------------------------------------
>                 Key: HADOOP-15003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15003
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, HADOOP-13786-043.patch,
HADOOP-13786-044.patch, HADOOP-13786-045.patch, HADOOP-13786-046.patch, HADOOP-13786-047.patch,
HADOOP-13786-048.patch, HADOOP-13786-049.patch, HADOOP-13786-050.patch, HADOOP-13786-051.patch,
> This is a Yetus only JIRA created to have Yetus review the HADOOP-13786/HADOOP-14971
patch as a .patch file, as the review PR [https://github.com/apache/hadoop/pull/282] is stopping
this happening in HADOOP-14971.
> Reviews should go into the PR/other task

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message