hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
Date Wed, 03 May 2017 14:03:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994929#comment-15994929

Daniel Templeton commented on HADOOP-13714:

{quote}We do have some logs whose output is considered stable: those machine readable ones
(HDFS Audit log &c). I'd like the explicit ones to be listed here with all others considered
open. We can't control the logging or error messages from downstream code either.{quote}

Totally agree.

Exceptions are covered by the API policy, and exception messages are covered by the log and
CLI policies.  Do we need a separate statement about exceptions?

{quote}If you look at the FS Spec, you can see that something a bit higher level works, ish.{quote}

The FileSystem spec is indeed very clear, and it would be awesome to have something like that
for all of Hadoop's interfaces, but we don't, and I'm not convinced that of our interfaces
can be specified that way.  I also don't think an end user is not going to read that doc.
 That's a doc for the developer community.  End users read javadoc.  Javadoc is also much
easier to create than a formal specification.  Yeah, it's not as formal as a formal specification,
but it's a tradeoff.

{quote}Consider also the tests to be our implicit specification{quote}

That sounds great in theory.  Our tests are far from comprehensive, though, and consider the
end user.  I, as a Hadoop developer, can't figure out what some of our tests are supposed
to be doing.  Do you really think an end user can?  Do you think they're even likely to try?
 If I want to know what something does, I read the source code for it, not the tests.  And
there's the issue.  Javadocs are cheap compared to writing formal specs or tests, they're
the first thing end users are going to look at, and when there are no javadocs end users turn
to reading the source code.  Once they're reading the source code to determine semantics,
we lose the ability to make even small changes without breaking things.

Of course, this discussion is largely moot, though, because I don't see the community investing
enough effort to create comprehensive tests, javadocs, or formal specs any time soon.

To [~aw]'s point, just writing a doc isn't enough.  The developer community needs to understand
what compatibility means and be committed to upholding it. Having a clear doc that covers
all the bases (we can think of) is a step in the right direction.  I don't think we can throw
up our hands and declare cross-release compatibility a quaint fiction of the past.  Adoption
of a platform like Hadoop depends on end users having a reliable and predictable build/integration

By the way, [~aw], what did you mean by Semantic Versioning?  Looking at http://semver.org/,
which would appear to be the canonical source, I see {quote}Given a version number MAJOR.MINOR.PATCH,
increment the:

1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backwards-compatible manner, and
3. PATCH version when you make backwards-compatible bug fixes.
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH
format.{quote}  Isn't that exactly what we already do and what this JIRA is attempting to
support?  What am I missing?

> Tighten up our compatibility guidelines for Hadoop 3
> ----------------------------------------------------
>                 Key: HADOOP-13714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13714
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 2.7.3
>            Reporter: Karthik Kambatla
>            Assignee: Daniel Templeton
>            Priority: Blocker
>         Attachments: HADOOP-13714.WIP-001.patch
> Our current compatibility guidelines are incomplete and loose. For many categories, we
do not have a policy. It would be nice to actually define those policies so our users know
what to expect and the developers know what releases to target their changes. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message