Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 16 Oct 2017 11:21:00 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13102746.1505546885000.4442.1508152860328@Atlassian.JIRA>
In-Reply-To: <JIRA.13102746.1505546885000@Atlassian.JIRA>
References: <JIRA.13102746.1505546885000@Atlassian.JIRA> <JIRA.13102746.1505546885471@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14876) Create downstream developer docs
 from the compatibility guidelines
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 16 Oct 2017 11:21:05 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205748#comment-16205748 ] 

Steve Loughran commented on HADOOP-14876:
-----------------------------------------

h4. Privacy scope

* add: sometimes things are marked as private when they end up being essential (example: UserGroupInformation). In situations,  raise issues with the team to see if we can't add some form of @public tag  \cite{HADOOP-10776}. 

Now, what about the fact that distributed shell example has (or at least did when I last looked) use of private code?
e.g org.apache.hadoop.io.DataOutputBuffer, the timeline plugin, NMClientAsyncImpl, ...  You can look at the imports and probably 20% of the class imports (not interfaces, yarn records) are tagged as private/limited private. We are not in a position to tell people not to use @Private, not given we consider doing so essential even for basic example yarn apps.

* What does it mean if something is tagged as LimitedPrivate for one app (esp HBase & Hive, which aren't within our own codebase)? to me, that says "we know these things get used downstream, or we've added them as a special secret back-door". But who gets to choose which apps can actually use it? Limited private+outside our codebase == public, which is something we should acknowledge when scoping things. And LimitedPrivate(Mapreduce) often means "every YARN app needs these".

* What does it mean if a release removes/changes something you depended on which was tagged private/limited private. Complain. It may get ignored, but it may have been done without awareness of wide use.

h4. Semantics

I take this bit very seriously, having been deeply involved in the original paragraphs, and an aficionado of all D.L. Parnas's writings on the notion of "interface".  As far as I'm concerned, the defacto definitions of semantics are defined in our unit tests "what we expect" and in those of widely used applications "what HBase and Hive expect". We know if we break the latter then people complain, and, while we may do so, its not something want to. B

L113. yeah right. It's usually the first port of call, & if you think otherwise, you're not writing enough downstream code. 

the original Compatibility.md calls out that some bits of the system have non-normative specifications; eg fileystem. I would consider that significantly more normative than the javadocs, most of which are vague aspirations of functionality. Usually the javadocs don't have any mention of concurrency, which matters a lot; for that you do end up delving into the source and/or using it in a way which appears to work (HDFS's use of input streams), when in fact they'r just using accidental bits of the semantics which we are now expected to maintain.


+maybe mention StreamCapabilities.hasCapability as a way of determining if FS streams offer a feature, say it's more to support variants in back ends rather than a way for us to remove things. But do mention: good practice to check for new things rather than assume that if HDFS implements it, it works everywhere.

L160" The audit log format may not change incompatibly between major releases." ?? "may change?" or "must not"

L189. Need to explain how to differentiate log chaff from "real" output. Indeed, I'm curious myself.


L208. We don't require log4j though; other back ends may be supportable.

L229. Nothing called "s3.*: no more

L298 "No new (exposed) dependency will be added to Hadoop between major releases."

Can't make that guarantee. Qualify "via the shaded clients"


Things that we've glossed over

* No statement on supported operating systems, filesystems, x86 parts IPv4 vs v6, If I code for Windows, how long will hadoop-client work there? What if I target SPARC?

* Concurrency: say "we try not to make things worse"?; degradations are considered defects except when its just some accidental side effect of excessive logging?


> Create downstream developer docs from the compatibility guidelines
> ------------------------------------------------------------------
>
>                 Key: HADOOP-14876
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14876
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 3.0.0-beta1
>            Reporter: Daniel Templeton
>            Assignee: Daniel Templeton
>            Priority: Critical
>         Attachments: HADOOP-14876.001.patch
>
>


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org