hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
Date Wed, 03 May 2017 18:40:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995399#comment-15995399

Steve Loughran commented on HADOOP-13714:

I care about compatibility; I think everyone does. it's just really hard to achieve.

regarding Semantic versioning, the problem is "all changes may break things". Even ifs something
as minor as changing a string in an exception message, something else could be looking for
it and find it breaks. Or subtle differences in performance and concurrency which look like
they work, but have adverse consequences downstream. I am very bleak about semantic versioning
being viable.

Now, regarding the specs, the FS spec was written precisely because too much of the FS behaviour
was hidden in the HDFS code and nobody had written down all that was happening, including
what the exceptions were, and there was nothing clear as to what features were deliberate
versus accidental (example: mkdirs -p /a/b/c being atomic. Deliberate? Or accidental side
effect of a locking optimisation? And what happens if it is now changed?). Its incomplete
(HADOOP-13327) and there's a tendency for new features in HDFS to consider it something unimportant,
leading to issues like HADOOP-14365. I understand why (timetable pressure, test centric dev
doesn't focus on the specs), but it's frustrating. 

To be fair though: it does get read, it is broadly understood by people, which shows how Python
makes a good syntax for specification.

Where it is limited programatically is that as it isn't something you can use in theorem provers,
the way TLA+ can be, I can't use it in some specification of what a committer does, use it
to prove that the MR committer V1 and v2 algorithms work, etc. I'm playing with the more rigorous
approach in HADOOP-13786, but I know once I have got a TLA+ spec for an object store &
its commit algorithm,. *nobody* is going to review it. We just don't have enough people who
play in that area to to the reviewing. Which means it wouldn't get the maintenance either.
Python it is, then.

> Tighten up our compatibility guidelines for Hadoop 3
> ----------------------------------------------------
>                 Key: HADOOP-13714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13714
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 2.7.3
>            Reporter: Karthik Kambatla
>            Assignee: Daniel Templeton
>            Priority: Blocker
>         Attachments: HADOOP-13714.WIP-001.patch
> Our current compatibility guidelines are incomplete and loose. For many categories, we
do not have a policy. It would be nice to actually define those policies so our users know
what to expect and the developers know what releases to target their changes. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message