hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12443) LocalDirAllocator shouldn't accept pathStr parameter with scheme or authority.
Date Sun, 27 Sep 2015 23:58:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909934#comment-14909934
] 

Hadoop QA commented on HADOOP-12443:
------------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  17m  4s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 1 new
or modified test files. |
| {color:green}+1{color} | javac |   7m 58s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |  10m 14s | There were no new javadoc warning messages.
|
| {color:green}+1{color} | release audit |   0m 24s | The applied patch does not increase
the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m  7s | The applied patch generated  1 new checkstyle
issues (total was 23, now 24). |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that end in whitespace.
|
| {color:green}+1{color} | install |   1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 34s | The patch built with eclipse:eclipse.
|
| {color:green}+1{color} | findbugs |   1m 56s | The patch does not introduce any new Findbugs
(version 3.0.0) warnings. |
| {color:red}-1{color} | common tests |   7m 50s | Tests failed in hadoop-common. |
| | |  48m 40s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12762621/HADOOP-12443.000.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 1c030c6 |
| checkstyle |  https://builds.apache.org/job/PreCommit-HADOOP-Build/7715/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/7715/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/7715/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/7715/console |


This message was automatically generated.

> LocalDirAllocator shouldn't accept pathStr parameter with scheme or authority.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-12443
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12443
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>         Attachments: HADOOP-12443.000.patch
>
>
> {{LocalDirAllocator}} shouldn't accept {{pathStr}} parameter with scheme or authority.
> Currently {{LocalDirAllocator}} accepts {{pathStr}} with scheme or authority, When {{pathStr}}
with scheme or authority is passed to {{getLocalPathForWrite}}, it will bypass {{localDirs}}
to use {{pathStr}} directly , then the return Path will be independent with {{localDirs}}.
> The reason is the following:
> {{LocalDirAllocator}} will use {{new Path(new Path(localDirs[dirNumLastAccessed]), pathStr)}}
as the return Path.
> The constructor code for {{Path}} is
> {code}
>   public Path(Path parent, Path child) {
>     // Add a slash to parent's path so resolution is compatible with URI's
>     URI parentUri = parent.uri;
>     String parentPath = parentUri.getPath();
>     if (!(parentPath.equals("/") || parentPath.isEmpty())) {
>       try {
>         parentUri = new URI(parentUri.getScheme(), parentUri.getAuthority(),
>                       parentUri.getPath()+"/", null, parentUri.getFragment());
>       } catch (URISyntaxException e) {
>         throw new IllegalArgumentException(e);
>       }
>     }
>     URI resolved = parentUri.resolve(child.uri);
>     initialize(resolved.getScheme(), resolved.getAuthority(),
>                resolved.getPath(), resolved.getFragment());
>   }
> {code}
> The above {{Path}} constructor code will call {{URI#resolve}} to merge the parent path
with child path.
> {code}
>     private static URI resolve(URI base, URI child) {
>         // check if child if opaque first so that NPE is thrown
>         // if child is null.
>         if (child.isOpaque() || base.isOpaque())
>             return child;
>         // 5.2 (2): Reference to current document (lone fragment)
>         if ((child.scheme == null) && (child.authority == null)
>             && child.path.equals("") && (child.fragment != null)
>             && (child.query == null)) {
>             if ((base.fragment != null)
>                 && child.fragment.equals(base.fragment)) {
>                 return base;
>             }
>             URI ru = new URI();
>             ru.scheme = base.scheme;
>             ru.authority = base.authority;
>             ru.userInfo = base.userInfo;
>             ru.host = base.host;
>             ru.port = base.port;
>             ru.path = base.path;
>             ru.fragment = child.fragment;
>             ru.query = base.query;
>             return ru;
>         }
>         // 5.2 (3): Child is absolute
>         if (child.scheme != null)
>             return child;
>         URI ru = new URI();             // Resolved URI
>         ru.scheme = base.scheme;
>         ru.query = child.query;
>         ru.fragment = child.fragment;
>         // 5.2 (4): Authority
>         if (child.authority == null) {
>             ru.authority = base.authority;
>             ru.host = base.host;
>             ru.userInfo = base.userInfo;
>             ru.port = base.port;
>             String cp = (child.path == null) ? "" : child.path;
>             if ((cp.length() > 0) && (cp.charAt(0) == '/')) {
>                 // 5.2 (5): Child path is absolute
>                 ru.path = child.path;
>             } else {
>                 // 5.2 (6): Resolve relative path
>                 ru.path = resolvePath(base.path, cp, base.isAbsolute());
>             }
>         } else {
>             ru.authority = child.authority;
>             ru.host = child.host;
>             ru.userInfo = child.userInfo;
>             ru.host = child.host;
>             ru.port = child.port;
>             ru.path = child.path;
>         }
>         // 5.2 (7): Recombine (nothing to do here)
>         return ru;
>     }
> {code}
> You can see if the child's uri has scheme or authority, it won't use anything from parent's
uri.
> This will hide the issue for user. For example, user passed file:///build/test/temp as
{{pathStr}} parameter to {{getLocalPathForWrite}}.
> Later on user may run into very strange problem: /build/test/temp directory is full because
return path is not from {{localDirs}}. This makes the issue very difficult for user to debug.
So it will be better to reject {{pathStr}} parameter with scheme or authority.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message