hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12666) Support Windows Azure Data Lake - as a file system in Hadoop
Date Mon, 01 Feb 2016 18:17:40 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126704#comment-15126704
] 

Hadoop QA commented on HADOOP-12666:
------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 6 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 42s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s {color} | {color:green}
trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s {color} |
{color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 33s {color}
| {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue}
Skipped branch modules with no Java source: hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s {color} |
{color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s {color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 35s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s {color} | {color:green}
the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s {color} |
{color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 40s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 46s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue}
Skipped patch modules with no Java source: hadoop-tools hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s {color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 58s {color} | {color:red}
hadoop-tools-jdk1.8.0_66 with JDK v1.8.0_66 generated 403 new + 403 unchanged - 0 fixed =
806 total (was 403) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 59s {color} | {color:red}
hadoop-tools-jdk1.8.0_66 with JDK v1.8.0_66 generated 403 new + 403 unchanged - 0 fixed =
806 total (was 403) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s {color} |
{color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m 20s {color} | {color:red}
hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 10 new + 10 unchanged - 0 fixed = 20
total (was 10) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m 20s {color} | {color:red}
hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 10 new + 10 unchanged - 0 fixed = 20
total (was 10) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 6s {color} | {color:green}
the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 31m 24s {color} | {color:green}
hadoop-tools in the patch passed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 30m 28s {color} | {color:green}
hadoop-tools in the patch passed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 32m 4s {color} | {color:green}
hadoop-tools in the patch passed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 17s {color} | {color:green}
hadoop-tools in the patch passed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 27s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12785489/HADOOP-12666-003.patch
|
| JIRA Issue | HADOOP-12666 |
| Optional Tests |  asflicense  findbugs  xml  compile  javac  javadoc  mvninstall  mvnsite
 unit  checkstyle  |
| uname | Linux f37c2d652e96 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8f2622b |
| Default Java | 1.7.0_91 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
|
| javadoc | hadoop-tools-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_66.txt
|
| javadoc | hadoop-tools-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_66.txt
|
| javadoc | hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt
|
| javadoc | hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt
|
| JDK v1.7.0_91  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/testReport/
|
| modules | C:  hadoop-tools   hadoop-tools/hadoop-azure-datalake  U: hadoop-tools |
| Max memory used | 77MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT   http://yetus.apache.org |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/console |


This message was automatically generated.



> Support Windows Azure Data Lake - as a file system in Hadoop
> ------------------------------------------------------------
>
>                 Key: HADOOP-12666
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12666
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs, fs/azure, tools
>            Reporter: Vishwajeet Dusane
>            Assignee: Vishwajeet Dusane
>         Attachments: HADOOP-12666-002.patch, HADOOP-12666-003.patch, HADOOP-12666-1.patch
>
>   Original Estimate: 336h
>          Time Spent: 336h
>  Remaining Estimate: 0h
>
> h2. Description
> This JIRA describes a new file system implementation for accessing Windows Azure Data
Lake Store (ADL) from within Hadoop. This would enable existing Hadoop applications such has
MR, HIVE, Hbase etc..,  to use ADL store as input or output.
>  
> ADL is ultra-high capacity, Optimized for massive throughput with rich management and
security features. More details available at https://azure.microsoft.com/en-us/services/data-lake-store/
> h2. High level design
> ADL file system exposes RESTful interfaces compatible with WebHdfs specification 2.7.1.
> At a high level, the code here extends the SWebHdfsFileSystem class to provide an implementation
for accessing ADL storage; the scheme ADL is used for accessing it over HTTPS. We use the
URI scheme:
> {code}adl://<URI to account>/path/to/file{code} 
> to address individual Files/Folders. Tests are implemented mostly using a Contract implementation
for the ADL functionality, with an option to test against a real ADL storage if configured.
> h2. Credits and history
> This has been ongoing work for a while, and the early version of this work can be seen
in. Credit for this work goes to the team: [~vishwajeet.dusane], [~snayak], [~srevanka], [~kiranch],
[~chakrab], [~omkarksa], [~snvijaya], [~ansaiprasanna]  [~jsangwan]
> h2. Test
> Besides Contract tests, we have used ADL as the additional file system in the current
public preview release. Various different customer and test workloads have been run against
clusters with such configurations for quite some time. The current version reflects to the
version of the code tested and used in our production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message