ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: [DISCUSS] Ignite Update Checker
Date Mon, 02 Oct 2017 09:46:37 GMT
I am not sure it is good idea to send requests to 3rd-party addresses from
Ignite node. Let's do not make the same mistakes again.

On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <anovikov@gridgain.com>
wrote:

> We may directly send request to GA from Ignite node:
> https://developers.google.com/analytics/devguides/collection/protocol/v1/
> <https://developers.google.com/analytics/devguides/collection/protocol/v1/
> >
> Latest version can be received from maven central:
> https://repo1.maven.org/maven2/org/apache/ignite/
> ignite-core/maven-metadata.xml <https://repo1.maven.org/
> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>
>
> > 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrakyan@apache.org>
> написал(а):
> >
> > Denis,
> >
> > I am not sure I understand. We already do have CGI enabled for
> > download.cgi. Is there something else we need?
> >
> > D.
> >
> > On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dmagda@gridgain.com> wrote:
> >
> >> There is an obstacle. There is no way to execute the script using PHP or
> >> similar sever side language and trigger GA as discussed earlier:
> >> https://issues.apache.org/jira/browse/INFRA-15182
> >>
> >> How else can we tackle this?
> >>
> >> Denis
> >>
> >> On Thursday, September 7, 2017, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> >> wrote:
> >>
> >>> I think it is safe to assume at this point that everyone is in general
> >>> agreement, since there are no active objections.
> >>>
> >>> I have filed a ticket for the 2.3 release. Let's try to make it happen:
> >>> https://issues.apache.org/jira/browse/IGNITE-6305
> >>>
> >>> D.
> >>>
> >>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> >> dsetrakyan@apache.org
> >>> <javascript:;>>
> >>> wrote:
> >>>
> >>>>
> >>>>
> >>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com
> >>> <javascript:;>>
> >>>> wrote:
> >>>>
> >>>>> Yeah, I guess that's doable as well and requires less management
> >> effort
> >>>>> than my suggestion. We could use events [1] to store payload data
> >> (e.g.
> >>>>> IP,
> >>>>> version, etc.)
> >>>>
> >>>>
> >>>> Yes, we could use events or some other similar API provided by GA.
> >>>>
> >>>>
> >>>>> What the download page CGI developed in? PHP?
> >>>>>
> >>>>
> >>>> To be honest, no clue. I guess someone in the community can figure it
> >>> out:
> >>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> >>>>
> >>>>
> >>>>> However, I'm not sure whether storing this data in a 3rd party
> >> (Google)
> >>> is
> >>>>> compliant with the ASF policy. I guess it's no biggie, but if there's
> >>>>> doubt
> >>>>> in the PMC, it's better to ask legal@.
> >>>>
> >>>>
> >>>> I am not sure there is anything to ask about. The whole Ignite website
> >> is
> >>>> GA enabled, and all we are doing is accessing a standard web page from
> >>> the
> >>>> Ignite web site. The information gathered from GA is available only
to
> >>> the
> >>>> Ignite PMC. Frankly, I think legal@ will be very confused by this
> >>>> question.
> >>>>
> >>>> Even ASF website itself uses GA: https://www.apache.org/
> >>>> foundation/policies/privacy.html
> >>>>
> >>>>
> >>>>> I think Cos said it's OK; maybe Roman can pitch in.
> >>>>>
> >>>>
> >>>> Sure, would be nice to hear from Roman as well.
> >>>>
> >>>>
> >>>>> Cheers.
> >>>>>
> >>>>> [1]
> >>>>> https://developers.google.com/analytics/devguides/collection
> >>>>> /analyticsjs/events
> >>>>>
> >>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> >>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>> wrote:
> >>>>>
> >>>>>> Raul,
> >>>>>>
> >>>>>> Could point about Javascript, it will not work because it executes
> >> in
> >>>>> the
> >>>>>> browser. This means we need a server-side script, like CGI we
are
> >>> using
> >>>>> on
> >>>>>> our download page.
> >>>>>>
> >>>>>> How about this approach. We create something like ignite-version.cgi
> >>>>> script
> >>>>>> which will invoke a call to GA and then return the latest version.
> >>>>>>
> >>>>>> This should work, right?
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> >> raul.asf@evosent.com
> >>> <javascript:;>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hey Dmitriy and all
> >>>>>>>
> >>>>>>> Also, since we have GA enabled for the website, we can track
how
> >>> many
> >>>>>> times
> >>>>>>>> this page was accessed, which will be equal to the number
of
> >>> starts.
> >>>>>> This
> >>>>>>>> way, the counter information is tracked and monitored
by the
> >>> Ignite
> >>>>>> PMC.
> >>>>>>>
> >>>>>>>
> >>>>>>> Unfortunately this won't work because GA is loaded via Javascript
> >> on
> >>>>> the
> >>>>>>> browser, so Google will never receive the page hit.
> >>>>>>>
> >>>>>>> Given the constraints, the most viable solution is an HTTPS
> >> endpoint
> >>>>>>> running on ASF infra that Ignite invokes via a GET or POST
> >> request.
> >>>>> The
> >>>>>>> simplest thing is to write each request in a log file, along
with
> >>> the
> >>>>>>> timestamp, the version reported by the client, maybe the
IP (not
> >>> sure
> >>>>>> about
> >>>>>>> the ASF rules about this concerning privacy, I guess it's
OK if
> >> you
> >>>>> make
> >>>>>> it
> >>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the
node
> >>> creates
> >>>>> on
> >>>>>>> first startup or something.
> >>>>>>>
> >>>>>>> This endpoint would need some basic DDoS protection and
> >> blacklisting
> >>>>> to
> >>>>>>> prevent data spoofing.
> >>>>>>>
> >>>>>>> If we'll be implementing this endpoint anyway, then there's
no
> >> point
> >>>>>>> placing another file on Git or elsewhere for reporting the
latest
> >>>>>> versions:
> >>>>>>> the endpoint itself can return them.
> >>>>>>>
> >>>>>>> WDYT?
> >>>>>>>
> >>>>>>> Cheers.
> >>>>>>>
> >>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> >>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Cos, Raul,
> >>>>>>>>
> >>>>>>>> Thanks for the feedback. I completely agree about Maven
Central
> >>>>> being a
> >>>>>>> 3rd
> >>>>>>>> party repo (did not think about that initially). All
your
> >>>>> suggestions
> >>>>>>> make
> >>>>>>>> sense, but I would like to keep it as simple as possible,
and so
> >>> far
> >>>>>>>> everything suggested required GIT dependencies and extra
work.
> >>>>>>>>
> >>>>>>>> How about Yakov's suggestion. We simply add a page to
the Ignite
> >>>>>> website
> >>>>>>>> which will have only the latest version. Every time
a node
> >> starts,
> >>>>> it
> >>>>>>>> receives the latest version from the page and suggests
that
> >> users
> >>>>>> upgrade
> >>>>>>>> if needed.
> >>>>>>>>
> >>>>>>>> Also, since we have GA enabled for the website, we can
track how
> >>>>> many
> >>>>>>> times
> >>>>>>>> this page was accessed, which will be equal to the number
of
> >>> starts.
> >>>>>> This
> >>>>>>>> way, the counter information is tracked and monitored
by the
> >>> Ignite
> >>>>>> PMC.
> >>>>>>>>
> >>>>>>>> This approach looks pretty innocent to me and everything
is kept
> >>> and
> >>>>>>>> managed within Apache.
> >>>>>>>>
> >>>>>>>> Thoughts?
> >>>>>>>>
> >>>>>>>> D.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik
<
> >>>>> cos@apache.org <javascript:;>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I agree with Raul.
> >>>>>>>>> - providing a ping-back address to a 3rd party might
be frown
> >>>>> upon by
> >>>>>>>> some.
> >>>>>>>>>  And might have a consequences like stats collection
about
> >>> users'
> >>>>>>>>>  infrastructure.
> >>>>>>>>> - checking an ASF git-repo is easy and won't download
any
> >> binary
> >>>>>> data:
> >>>>>>>>>  everything is clear text and could be easily monitored
by
> >> any
> >>> of
> >>>>>> the
> >>>>>>>>> network
> >>>>>>>>>  diagnostic tools, shall it be required. But it
involves a
> >> bit
> >>> of
> >>>>>> the
> >>>>>>>>> release
> >>>>>>>>>  discipline.
> >>>>>>>>> - the binary data download in the runtime is my
main concern.
> >>>>> This is
> >>>>>>> the
> >>>>>>>>>  vector of MMA. What if someone gains the control
over the
> >>>>>> repository
> >>>>>>>> and
> >>>>>>>>>  replaces the file with some malicious content.
> >>>>>>>>>
> >>>>>>>>> As for the particular mechanism: IIRC Ignite used
to make a
> >> call
> >>>>> to
> >>>>>> an
> >>>>>>>>> external script to check upon the atest software
version
> >>> available
> >>>>>> for
> >>>>>>>>> download. In the past, the endpoint was running
on a 3rd party
> >>>>>> server,
> >>>>>>> I
> >>>>>>>>> believe the best way would be to put this script
on ASF infra
> >>> and
> >>>>>> have
> >>>>>>>> the
> >>>>>>>>> "update checker" running in a completely controlled
> >> environment.
> >>>>>>>> Actually,
> >>>>>>>>> I
> >>>>>>>>> recall we had this very discussion during the Incubation;
I
> >> can
> >>>>>>> probably
> >>>>>>>>> dig
> >>>>>>>>> out the corresponding thread.
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>  Cok
> >>>>>>>>>
> >>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani
wrote:
> >>>>>>>>>> Hey guys
> >>>>>>>>>>
> >>>>>>>>>> In my opinion, maven.org is still owned by a
third party
> >>>>>> (Sonatype).
> >>>>>>>> We
> >>>>>>>>>> don't know what kind of data analysis or intelligence
> >>> extraction
> >>>>>> they
> >>>>>>>>> run.
> >>>>>>>>>>
> >>>>>>>>>> If Ignite servers all over the world were hitting
maven.org
> >>>>>>>> periodically
> >>>>>>>>>> asking for an Ignite Maven artifact, it gives
Sonatype a
> >> clear
> >>>>>>>> indication
> >>>>>>>>>> of who is running an Ignite server.
> >>>>>>>>>>
> >>>>>>>>>> They could reverse lookup the IP address and
find out what
> >>>>>>> corporation
> >>>>>>>> it
> >>>>>>>>>> is.
> >>>>>>>>>>
> >>>>>>>>>> How about having Ignite check the ASF Git directly?
> >>>>>>>>>>
> >>>>>>>>>> We could use the Git ssh API, but that would
require a new
> >>>>>>> dependency,
> >>>>>>>>>> which I advise against.
> >>>>>>>>>>
> >>>>>>>>>> Alternatively, we could have it scrape this
HTML for new Git
> >>>>> tags:
> >>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> >>>>>>>>>>
> >>>>>>>>>> Another option is to place a txt file in the
root of the
> >>> master
> >>>>>>> branch
> >>>>>>>>> (e.g
> >>>>>>>>>> LATEST), containing a list of the latest GA
versions for
> >> each
> >>>>> major
> >>>>>>>>> version
> >>>>>>>>>> line (1.x, 2.x).
> >>>>>>>>>>
> >>>>>>>>>> I would advocate this last option, but it requires
somebody
> >>>>>>> remembering
> >>>>>>>>> to
> >>>>>>>>>> update the file with every release, unless we
automate it
> >>> with a
> >>>>>>> Maven
> >>>>>>>>>> plugin.
> >>>>>>>>>>
> >>>>>>>>>> Hope that helps!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> >>> <javascript:;>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I see nothing wrong with this approach.
> >>>>>>>>>>
> >>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do
you think? Is
> >> it
> >>>>> good
> >>>>>>> to
> >>>>>>>>> go?
> >>>>>>>>>>
> >>>>>>>>>> —
> >>>>>>>>>> Denis
> >>>>>>>>>>
> >>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan
<
> >>>>>>>> dsetrakyan@apache.org <javascript:;>
> >>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Is everyone OK with this approach? Should
I file a ticket
> >> on
> >>>>> it?
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy
Setrakyan <
> >>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>
> >>>>>>>>>>>> There has been lots of talk of proposals
about various
> >>> usage
> >>>>>>> metrics
> >>>>>>>>> for
> >>>>>>>>>>>> Ignite and nothing came of it. I would
like to resurrect
> >>> the
> >>>>>> topic
> >>>>>>>> and
> >>>>>>>>>>>> propose something very simple and non-intrusive.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Update Checker
> >>>>>>>>>>>> The main purpose of the update checker
is not to collect
> >>>>>> metrics,
> >>>>>>>> but
> >>>>>>>>> to
> >>>>>>>>>>>> notify users about a new version of
Ignite by accessing
> >>>>>> maven.org
> >>>>>>>> and
> >>>>>>>>>>>> getting the version out of the metadata
file:
> >>>>>>>>>>>> http://repo2.maven.org/maven2/
> >>> org/apache/ignite/ignite-core/
> >>>>>>>>>>>> maven-metadata.xml
> >>>>>>>>>>>>
> >>>>>>>>>>>> This way we do not send any information
anywhere and, at
> >>> the
> >>>>>> same
> >>>>>>>>> time,
> >>>>>>>>>>>> urge our users to download and start
using the latest
> >>>>> version of
> >>>>>>>>> Ignite.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. Startup Counter
> >>>>>>>>>>>> This piece is optional, but we can also
get an insight in
> >>> how
> >>>>>> many
> >>>>>>>>> times
> >>>>>>>>>> a
> >>>>>>>>>>>> certain Ignite release gets started.
This is just a cool
> >>>>> metric
> >>>>>>> for
> >>>>>>>>> the
> >>>>>>>>>>>> community to gauge the project popularity.
You can think
> >> of
> >>>>> it
> >>>>>> as
> >>>>>>>> of a
> >>>>>>>>>> page
> >>>>>>>>>>>> visit counter shown on many websites.
We can even decide
> >> to
> >>>>>>> display
> >>>>>>>>> this
> >>>>>>>>>>>> counter on the Ignite website as well.
> >>>>>>>>>>>>
> >>>>>>>>>>>> To do this, we can simply add a JAR
in maven for every
> >>>>> release,
> >>>>>>> e.g.
> >>>>>>>>>>>> ignite-start-counter.jar, which will
contain only 1 byte.
> >>>>> Every
> >>>>>>> time
> >>>>>>>>> an
> >>>>>>>>>>>> Ignite node starts, it will download
this JAR in the
> >>>>> background.
> >>>>>>>> Then
> >>>>>>>>> we
> >>>>>>>>>>>> will be able to view the number of the
total downloads
> >> for
> >>>>> this
> >>>>>>> JAR
> >>>>>>>> in
> >>>>>>>>>>>> Maven Central, which is essentially
the number of starts
> >> of
> >>>>>> Ignite
> >>>>>>>>> nodes.
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Note that neither of the above suggestions
require
> >> Ignite
> >>> to
> >>>>>> send
> >>>>>>>> or
> >>>>>>>>>>>> track any user information whatsoever.*
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please reply suggesting weather you
are OK with this
> >>>>> approach.
> >>>>>>>>>>>>
> >>>>>>>>>>>> D.
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message