hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14444) New implementation of ftp and sftp filesystems
Date Tue, 01 Aug 2017 16:20:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109195#comment-16109195

Steve Loughran commented on HADOOP-14444:


line endings: 
* I don't like 80 chars either, when writing and reviewing code I'm happy for the limit to
be relaxed in the interest of readability (i.e to avoid needless line breaks which make things
* I think it dates from punched cards. But it does aid side-by-side review in IDEs and browsers.
Personally I'd like 100-120, and do bring up the topic, but it gets declined. Sorry.

private variables. 

* For {{final}} stuff and small inline classes (which surface a lot in java 8 lambda-expressions),
the need to wrap all access is a PITA. For public production code though, aids maintenance.
* regarding java default privacy scope. Package-private is pretty odd; things would have been
better with private as the default, maybe even explicit property get/set mechanism as C# does.
But we don't, so instead we have machines to tell us off.

indentation &c
* It's about having something consistent between bits of the code for easier cut/paste &
patch merge. But do note that checkstyle falls over with lambda-expressions.

Regarding how well we adhere to the checkstyle complaints, if you look at HADOOP-13786 we
can see me having to deal with them. I split it them into "issues I can fix" and "things it
doesn't realise are needed" and fix the little ones —then argue in the JIRA why the others
should get in.

> New implementation of ftp and sftp filesystems
> ----------------------------------------------
>                 Key: HADOOP-14444
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14444
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 2.8.0
>            Reporter: Lukas Waldmann
>            Assignee: Lukas Waldmann
>         Attachments: HADOOP-14444.2.patch, HADOOP-14444.3.patch, HADOOP-14444.4.patch,
HADOOP-14444.5.patch, HADOOP-14444.6.patch, HADOOP-14444.7.patch, HADOOP-14444.8.patch, HADOOP-14444.patch
> Current implementation of FTP and SFTP filesystems have severe limitations and performance
issues when dealing with high number of files. Mine patch solve those issues and integrate
both filesystems such a way that most of the core functionality is common for both and therefore
simplifying the maintainability.
> The core features:
> * Support for HTTP/SOCKS proxies
> * Support for passive FTP
> * Support of connection pooling - new connection is not created for every single command
but reused from the pool.
> For huge number of files it shows order of magnitude performance improvement over not
pooled connections.
> * Caching of directory trees. For ftp you always need to list whole directory whenever
you ask information about particular file.
> Again for huge number of files it shows order of magnitude performance improvement over
not cached connections.
> * Support of keep alive (NOOP) messages to avoid connection drops
> * Support for Unix style or regexp wildcard glob - useful for listing a particular files
across whole directory tree
> * Support for reestablishing broken ftp data transfers - can happen surprisingly often

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message