Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <EA0B6423-C050-4597-9C69-95139334BFDF@hortonworks.com>
References: <732DB60A-844D-4FA3-822B-08DB587DEB9B@apache.org>
 <6192A234-0F2E-4CB4-B3C7-142728E0800A@effectivemachines.com>
 <A7EF3127-1DED-46F0-9AA4-A39A3F8DAC98@apache.org> <1471528833445.31857@hortonworks.com>
 <DC215C6F-413A-47C2-94B9-564C27FF446D@gmail.com> <EA0B6423-C050-4597-9C69-95139334BFDF@hortonworks.com>
From: Andrew Purtell <andrew.purtell@gmail.com>
Date: Thu, 18 Aug 2016 10:37:00 -0700
Message-ID: <CA+RK=_AWXMsM-9JUbgRQo2B8Ag4Q=CvUvuf8789yTAF0uQvKsg@mail.gmail.com>
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
To: Chris Nauroth <cnauroth@hortonworks.com>
Cc: Junping Du <jdu@hortonworks.com>, Vinod Kumar Vavilapalli <vinodkv@apache.org>,
	Allen Wittenauer <aw@effectivemachines.com>,
	"common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
	"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
	"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a1142bfacec3af2053a5c0965
archived-at: Thu, 18 Aug 2016 17:37:51 -0000

--001a1142bfacec3af2053a5c0965
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

>
What is a realistic strategy for us to evolve the HDFS audit log in a
backward-compatible way?  If the API is essentially any form of ad-hoc
scripting, then for any proposed audit log format change, I can find a
reason to veto it on grounds of backward incompatibility.

Yeah when log scraping is the only way at information, then the API surface
expands to cover all manner of ad-hoc scripting.

Not sure moving away from emitting audit information in log lines would be
operator friendly. That's a tough one. Just about everything in the
ecosystem emits audit information as log lines. If Hadoop switches strategy
to become a one-off doing something different this would be painful.

Assuming log lines will be the way we continue to receive audit events from
Hadoop/HDFS, please consider freezing any changes to audit logging today,
develop a formal specification, add the specification to documentation, and
then take care to not break the specification between releases. Because
audit logging from the NN comes from low level places in FSNameSystem this
is going to constrain maintenance and refactor of that and related code, so
with my software maintainer hat on I feel your pain in advance. You'll want
to hash out what level of compatibility you'd like to offer. I'd recommend
only changing on major releases.

On Thu, Aug 18, 2016 at 10:04 AM, Chris Nauroth <cnauroth@hortonworks.com>
wrote:

> Andrew, thanks for adding your perspective on this.
>
> =E2=80=8B=E2=80=8B
> What is a realistic strategy for us to evolve the HDFS audit log in a
> backward-compatible way?  If the API is essentially any form of ad-hoc
> scripting, then for any proposed audit log format change, I can find a
> reason to veto it on grounds of backward incompatibility.
>
> - I can=E2=80=99t add a new field on the end, because that would break an=
 awk
> script that uses $NF expecting to find a specific field.
> - I can=E2=80=99t prepend a new field, because that would break a "cut -f=
1"
> expecting to find the timestamp.
> - HDFS can=E2=80=99t add any new features, because someone might have wri=
tten a
> script that does "exit 1" if it finds an unexpected RPC in the "cmd=3D" f=
ield.
> - Hadoop is not allowed to add full IPv6 support, because someone might
> have written a script that looks at the "ip=3D" field and parses it by IP=
v4
> syntax.
>
> On the CLI, a potential solution for evolving the output is to preserve
> the old format by default and only enable the new format if the user
> explicitly passes a new argument.  What should we do for the audit log?
> Configuration flags in hdfs-site.xml?  (That of course adds its own brand
> of complexity.)
>
> =E2=80=8B=E2=80=8B
> I=E2=80=99m particularly interested to hear potential solutions from peop=
le like
> Andrew and Allen who have been most vocal about the need for a stable
> format.  Without a solution, this unfortunately devolves into the format
> being frozen within a major release line.
>
> We could benefit from getting a patch on the compatibility doc that
> addresses the HDFS audit log specifically.
>
> --Chris Nauroth
>
> On 8/18/16, 8:47 AM, "Andrew Purtell" <andrew.purtell@gmail.com> wrote:
>
>     An incompatible APIs change is developer unfriendly. An incompatible
> behavioral change is operator unfriendly. Historically, one dimension of
> incompatibility has had a lot more mindshare than the other. It's great
> that this might be changing for the better.
>
>     Where I work when we move from one Hadoop 2.x minor to another we
> always spend time updating our deployment plans, alerting, log scraping,
> and related things due to changes. Some are debatable as if qualifying fo=
r
> the 'incompatible' designation. I think the audit logging change that
> triggered this discussion is a good example of one that does. If you want
> to audit HDFS actions those log emissions are your API. (Inotify doesn't
> offer access control events.) One has to code regular expressions for
> parsing them and reverse engineer under what circumstances an audit line =
is
> emitted so you can make assumptions about what transpired. Change either
> and you might break someone's automation for meeting industry or legal
> compliance obligations. Not a trivial matter. If you don't operate Hadoop
> in production you might not realize the implications of such a change. Gl=
ad
> to see Hadoop has community diversity to recognize it in some cases.
>
>     > On Aug 18, 2016, at 6:57 AM, Junping Du <jdu@hortonworks.com> wrote=
:
>     >
>     > I think Allen's previous comments are very misleading.
>     > In my understanding, only incompatible API (RPC, CLIs, WebService,
> etc.) shouldn't land on branch-2, but other incompatible behaviors (logs,
> audit-log, daemon's restart, etc.) should get flexible for landing.
> Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with
> incompatible-changes could get landed on branch-2 after 2.2.0 release? Mo=
st
> of them are already released.
>     >
>     > Thanks,
>     >
>     > Junping
>     > ________________________________________
>     > From: Vinod Kumar Vavilapalli <vinodkv@apache.org>
>     > Sent: Wednesday, August 17, 2016 9:29 PM
>     > To: Allen Wittenauer
>     > Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>     > Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>     >
>     > I always look at CHANGES.txt entries for incompatible-changes and
> this JIRA obviously wasn=E2=80=99t there.
>     >
>     > Anyways, this shouldn=E2=80=99t be in any of branch-2.* as committe=
rs there
> clearly mentioned that this is an incompatible change.
>     >
>     > I am reverting the patch from branch-2* .
>     >
>     > Thanks
>     > +Vinod
>     >
>     >> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <
> aw@effectivemachines.com> wrote:
>     >>
>     >>
>     >>
>     >> -1
>     >>
>     >> HDFS-9395 is an incompatible change:
>     >>
>     >> a) Why is not marked as such in the changes file?
>     >> b) Why is an incompatible change in a micro release, much less a
> minor?
>     >> c) Where is the release note for this change?
>     >>
>     >>
>     >>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
>     >>>
>     >>> Hi all,
>     >>>
>     >>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>     >>>
>     >>> As discussed before, this is the next maintenance release to
> follow up 2.7.2.
>     >>>
>     >>> The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC0/>
>     >>>
>     >>> The RC tag in git is: release-2.7.3-RC1
>     >>>
>     >>> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1045/ <https://repository.apache.
> org/content/repositories/orgapachehadoop-1045/>
>     >>>
>     >>> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.htm=
l
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for your quick perusal.
>     >>>
>     >>> As you may have noted,
>     >>> - few issues with RC0 forced a RC1 [1]
>     >>> - a very long fix-cycle for the License & Notice issues
> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to sl=
ip
> by quite a bit. This release's related discussion thread is linked below:
> [2].
>     >>>
>     >>> Please try the release and vote; the vote will run for the usual =
5
> days.
>     >>>
>     >>> Thanks,
>     >>> Vinod
>     >>>
>     >>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#2610=
6
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/
> index.html#26106>
>     >>> [2]: 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
> 6yv2fyrs4jlepmmr>
>     >>
>     >>
>     >> ------------------------------------------------------------
> ---------
>     >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>     >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>     >
>     >
>     > ------------------------------------------------------------
> ---------
>     > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>     > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>     >
>     >
>     > ------------------------------------------------------------
> ---------
>     > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>     > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>     >
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>     For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
>
>
>

--001a1142bfacec3af2053a5c0965--