hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Date Thu, 18 Aug 2016 17:04:07 GMT
Andrew, thanks for adding your perspective on this.

What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible
way?  If the API is essentially any form of ad-hoc scripting, then for any proposed audit
log format change, I can find a reason to veto it on grounds of backward incompatibility.

- I can’t add a new field on the end, because that would break an awk script that uses $NF
expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find the
timestamp.
- HDFS can’t add any new features, because someone might have written a script that does
"exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script
that looks at the "ip=" field and parses it by IPv4 syntax.

On the CLI, a potential solution for evolving the output is to preserve the old format by
default and only enable the new format if the user explicitly passes a new argument.  What
should we do for the audit log?  Configuration flags in hdfs-site.xml?  (That of course adds
its own brand of complexity.)

I’m particularly interested to hear potential solutions from people like Andrew and Allen
who have been most vocal about the need for a stable format.  Without a solution, this unfortunately
devolves into the format being frozen within a major release line.

We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit
log specifically. 

--Chris Nauroth

On 8/18/16, 8:47 AM, "Andrew Purtell" <andrew.purtell@gmail.com> wrote:

    An incompatible APIs change is developer unfriendly. An incompatible behavioral change
is operator unfriendly. Historically, one dimension of incompatibility has had a lot more
mindshare than the other. It's great that this might be changing for the better. 
    
    Where I work when we move from one Hadoop 2.x minor to another we always spend time updating
our deployment plans, alerting, log scraping, and related things due to changes. Some are
debatable as if qualifying for the 'incompatible' designation. I think the audit logging change
that triggered this discussion is a good example of one that does. If you want to audit HDFS
actions those log emissions are your API. (Inotify doesn't offer access control events.) One
has to code regular expressions for parsing them and reverse engineer under what circumstances
an audit line is emitted so you can make assumptions about what transpired. Change either
and you might break someone's automation for meeting industry or legal compliance obligations.
Not a trivial matter. If you don't operate Hadoop in production you might not realize the
implications of such a change. Glad to see Hadoop has community diversity to recognize it
in some cases. 
    
    > On Aug 18, 2016, at 6:57 AM, Junping Du <jdu@hortonworks.com> wrote:
    > 
    > I think Allen's previous comments are very misleading. 
    > In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't
land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.)
should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5)
marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of
them are already released. 
    > 
    > Thanks,
    > 
    > Junping
    > ________________________________________
    > From: Vinod Kumar Vavilapalli <vinodkv@apache.org>
    > Sent: Wednesday, August 17, 2016 9:29 PM
    > To: Allen Wittenauer
    > Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
mapreduce-dev@hadoop.apache.org
    > Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
    > 
    > I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously
wasn’t there.
    > 
    > Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned
that this is an incompatible change.
    > 
    > I am reverting the patch from branch-2* .
    > 
    > Thanks
    > +Vinod
    > 
    >> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw@effectivemachines.com>
wrote:
    >> 
    >> 
    >> 
    >> -1
    >> 
    >> HDFS-9395 is an incompatible change:
    >> 
    >> a) Why is not marked as such in the changes file?
    >> b) Why is an incompatible change in a micro release, much less a minor?
    >> c) Where is the release note for this change?
    >> 
    >> 
    >>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org>
wrote:
    >>> 
    >>> Hi all,
    >>> 
    >>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
    >>> 
    >>> As discussed before, this is the next maintenance release to follow up 2.7.2.
    >>> 
    >>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/
<http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
    >>> 
    >>> The RC tag in git is: release-2.7.3-RC1
    >>> 
    >>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/>
at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
    >>> 
    >>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html.
I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
for your quick perusal.
    >>> 
    >>> As you may have noted,
    >>> - few issues with RC0 forced a RC1 [1]
    >>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's
related discussion thread is linked below: [2].
    >>> 
    >>> Please try the release and vote; the vote will run for the usual 5 days.
    >>> 
    >>> Thanks,
    >>> Vinod
    >>> 
    >>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
<https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
    >>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
<http://markmail.org/thread/6yv2fyrs4jlepmmr>
    >> 
    >> 
    >> ---------------------------------------------------------------------
    >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
    >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
    > 
    > 
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
    > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
    > 
    > 
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
    > For additional commands, e-mail: common-dev-help@hadoop.apache.org
    > 
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
    For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
    
    
    

Mime
View raw message