ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Novikov <anovi...@gridgain.com>
Subject Re: [DISCUSS] Ignite Update Checker
Date Mon, 02 Oct 2017 09:42:17 GMT
We may directly send request to GA from Ignite node: https://developers.google.com/analytics/devguides/collection/protocol/v1/
<https://developers.google.com/analytics/devguides/collection/protocol/v1/>
Latest version can be received from maven central: https://repo1.maven.org/maven2/org/apache/ignite/ignite-core/maven-metadata.xml
<https://repo1.maven.org/maven2/org/apache/ignite/ignite-core/maven-metadata.xml>


> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrakyan@apache.org> написал(а):
> 
> Denis,
> 
> I am not sure I understand. We already do have CGI enabled for
> download.cgi. Is there something else we need?
> 
> D.
> 
> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dmagda@gridgain.com> wrote:
> 
>> There is an obstacle. There is no way to execute the script using PHP or
>> similar sever side language and trigger GA as discussed earlier:
>> https://issues.apache.org/jira/browse/INFRA-15182
>> 
>> How else can we tackle this?
>> 
>> Denis
>> 
>> On Thursday, September 7, 2017, Dmitriy Setrakyan <dsetrakyan@apache.org>
>> wrote:
>> 
>>> I think it is safe to assume at this point that everyone is in general
>>> agreement, since there are no active objections.
>>> 
>>> I have filed a ticket for the 2.3 release. Let's try to make it happen:
>>> https://issues.apache.org/jira/browse/IGNITE-6305
>>> 
>>> D.
>>> 
>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>> dsetrakyan@apache.org
>>> <javascript:;>>
>>> wrote:
>>> 
>>>> 
>>>> 
>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com
>>> <javascript:;>>
>>>> wrote:
>>>> 
>>>>> Yeah, I guess that's doable as well and requires less management
>> effort
>>>>> than my suggestion. We could use events [1] to store payload data
>> (e.g.
>>>>> IP,
>>>>> version, etc.)
>>>> 
>>>> 
>>>> Yes, we could use events or some other similar API provided by GA.
>>>> 
>>>> 
>>>>> What the download page CGI developed in? PHP?
>>>>> 
>>>> 
>>>> To be honest, no clue. I guess someone in the community can figure it
>>> out:
>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>>>> 
>>>> 
>>>>> However, I'm not sure whether storing this data in a 3rd party
>> (Google)
>>> is
>>>>> compliant with the ASF policy. I guess it's no biggie, but if there's
>>>>> doubt
>>>>> in the PMC, it's better to ask legal@.
>>>> 
>>>> 
>>>> I am not sure there is anything to ask about. The whole Ignite website
>> is
>>>> GA enabled, and all we are doing is accessing a standard web page from
>>> the
>>>> Ignite web site. The information gathered from GA is available only to
>>> the
>>>> Ignite PMC. Frankly, I think legal@ will be very confused by this
>>>> question.
>>>> 
>>>> Even ASF website itself uses GA: https://www.apache.org/
>>>> foundation/policies/privacy.html
>>>> 
>>>> 
>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>>>>> 
>>>> 
>>>> Sure, would be nice to hear from Roman as well.
>>>> 
>>>> 
>>>>> Cheers.
>>>>> 
>>>>> [1]
>>>>> https://developers.google.com/analytics/devguides/collection
>>>>> /analyticsjs/events
>>>>> 
>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>> wrote:
>>>>> 
>>>>>> Raul,
>>>>>> 
>>>>>> Could point about Javascript, it will not work because it executes
>> in
>>>>> the
>>>>>> browser. This means we need a server-side script, like CGI we are
>>> using
>>>>> on
>>>>>> our download page.
>>>>>> 
>>>>>> How about this approach. We create something like ignite-version.cgi
>>>>> script
>>>>>> which will invoke a call to GA and then return the latest version.
>>>>>> 
>>>>>> This should work, right?
>>>>>> 
>>>>>> D.
>>>>>> 
>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>> raul.asf@evosent.com
>>> <javascript:;>>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hey Dmitriy and all
>>>>>>> 
>>>>>>> Also, since we have GA enabled for the website, we can track
how
>>> many
>>>>>> times
>>>>>>>> this page was accessed, which will be equal to the number
of
>>> starts.
>>>>>> This
>>>>>>>> way, the counter information is tracked and monitored by
the
>>> Ignite
>>>>>> PMC.
>>>>>>> 
>>>>>>> 
>>>>>>> Unfortunately this won't work because GA is loaded via Javascript
>> on
>>>>> the
>>>>>>> browser, so Google will never receive the page hit.
>>>>>>> 
>>>>>>> Given the constraints, the most viable solution is an HTTPS
>> endpoint
>>>>>>> running on ASF infra that Ignite invokes via a GET or POST
>> request.
>>>>> The
>>>>>>> simplest thing is to write each request in a log file, along
with
>>> the
>>>>>>> timestamp, the version reported by the client, maybe the IP (not
>>> sure
>>>>>> about
>>>>>>> the ASF rules about this concerning privacy, I guess it's OK
if
>> you
>>>>> make
>>>>>> it
>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node
>>> creates
>>>>> on
>>>>>>> first startup or something.
>>>>>>> 
>>>>>>> This endpoint would need some basic DDoS protection and
>> blacklisting
>>>>> to
>>>>>>> prevent data spoofing.
>>>>>>> 
>>>>>>> If we'll be implementing this endpoint anyway, then there's no
>> point
>>>>>>> placing another file on Git or elsewhere for reporting the latest
>>>>>> versions:
>>>>>>> the endpoint itself can return them.
>>>>>>> 
>>>>>>> WDYT?
>>>>>>> 
>>>>>>> Cheers.
>>>>>>> 
>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Cos, Raul,
>>>>>>>> 
>>>>>>>> Thanks for the feedback. I completely agree about Maven Central
>>>>> being a
>>>>>>> 3rd
>>>>>>>> party repo (did not think about that initially). All your
>>>>> suggestions
>>>>>>> make
>>>>>>>> sense, but I would like to keep it as simple as possible,
and so
>>> far
>>>>>>>> everything suggested required GIT dependencies and extra
work.
>>>>>>>> 
>>>>>>>> How about Yakov's suggestion. We simply add a page to the
Ignite
>>>>>> website
>>>>>>>> which will have only the latest version. Every time a node
>> starts,
>>>>> it
>>>>>>>> receives the latest version from the page and suggests that
>> users
>>>>>> upgrade
>>>>>>>> if needed.
>>>>>>>> 
>>>>>>>> Also, since we have GA enabled for the website, we can track
how
>>>>> many
>>>>>>> times
>>>>>>>> this page was accessed, which will be equal to the number
of
>>> starts.
>>>>>> This
>>>>>>>> way, the counter information is tracked and monitored by
the
>>> Ignite
>>>>>> PMC.
>>>>>>>> 
>>>>>>>> This approach looks pretty innocent to me and everything
is kept
>>> and
>>>>>>>> managed within Apache.
>>>>>>>> 
>>>>>>>> Thoughts?
>>>>>>>> 
>>>>>>>> D.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
>>>>> cos@apache.org <javascript:;>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I agree with Raul.
>>>>>>>>> - providing a ping-back address to a 3rd party might
be frown
>>>>> upon by
>>>>>>>> some.
>>>>>>>>>  And might have a consequences like stats collection
about
>>> users'
>>>>>>>>>  infrastructure.
>>>>>>>>> - checking an ASF git-repo is easy and won't download
any
>> binary
>>>>>> data:
>>>>>>>>>  everything is clear text and could be easily monitored
by
>> any
>>> of
>>>>>> the
>>>>>>>>> network
>>>>>>>>>  diagnostic tools, shall it be required. But it involves
a
>> bit
>>> of
>>>>>> the
>>>>>>>>> release
>>>>>>>>>  discipline.
>>>>>>>>> - the binary data download in the runtime is my main
concern.
>>>>> This is
>>>>>>> the
>>>>>>>>>  vector of MMA. What if someone gains the control over
the
>>>>>> repository
>>>>>>>> and
>>>>>>>>>  replaces the file with some malicious content.
>>>>>>>>> 
>>>>>>>>> As for the particular mechanism: IIRC Ignite used to
make a
>> call
>>>>> to
>>>>>> an
>>>>>>>>> external script to check upon the atest software version
>>> available
>>>>>> for
>>>>>>>>> download. In the past, the endpoint was running on a
3rd party
>>>>>> server,
>>>>>>> I
>>>>>>>>> believe the best way would be to put this script on ASF
infra
>>> and
>>>>>> have
>>>>>>>> the
>>>>>>>>> "update checker" running in a completely controlled
>> environment.
>>>>>>>> Actually,
>>>>>>>>> I
>>>>>>>>> recall we had this very discussion during the Incubation;
I
>> can
>>>>>>> probably
>>>>>>>>> dig
>>>>>>>>> out the corresponding thread.
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>>  Cok
>>>>>>>>> 
>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>>>>>>>>>> Hey guys
>>>>>>>>>> 
>>>>>>>>>> In my opinion, maven.org is still owned by a third
party
>>>>>> (Sonatype).
>>>>>>>> We
>>>>>>>>>> don't know what kind of data analysis or intelligence
>>> extraction
>>>>>> they
>>>>>>>>> run.
>>>>>>>>>> 
>>>>>>>>>> If Ignite servers all over the world were hitting
maven.org
>>>>>>>> periodically
>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype
a
>> clear
>>>>>>>> indication
>>>>>>>>>> of who is running an Ignite server.
>>>>>>>>>> 
>>>>>>>>>> They could reverse lookup the IP address and find
out what
>>>>>>> corporation
>>>>>>>> it
>>>>>>>>>> is.
>>>>>>>>>> 
>>>>>>>>>> How about having Ignite check the ASF Git directly?
>>>>>>>>>> 
>>>>>>>>>> We could use the Git ssh API, but that would require
a new
>>>>>>> dependency,
>>>>>>>>>> which I advise against.
>>>>>>>>>> 
>>>>>>>>>> Alternatively, we could have it scrape this HTML
for new Git
>>>>> tags:
>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>>>>>>>> 
>>>>>>>>>> Another option is to place a txt file in the root
of the
>>> master
>>>>>>> branch
>>>>>>>>> (e.g
>>>>>>>>>> LATEST), containing a list of the latest GA versions
for
>> each
>>>>> major
>>>>>>>>> version
>>>>>>>>>> line (1.x, 2.x).
>>>>>>>>>> 
>>>>>>>>>> I would advocate this last option, but it requires
somebody
>>>>>>> remembering
>>>>>>>>> to
>>>>>>>>>> update the file with every release, unless we automate
it
>>> with a
>>>>>>> Maven
>>>>>>>>>> plugin.
>>>>>>>>>> 
>>>>>>>>>> Hope that helps!
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
>>> <javascript:;>> wrote:
>>>>>>>>>> 
>>>>>>>>>> I see nothing wrong with this approach.
>>>>>>>>>> 
>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you
think? Is
>> it
>>>>> good
>>>>>>> to
>>>>>>>>> go?
>>>>>>>>>> 
>>>>>>>>>> —
>>>>>>>>>> Denis
>>>>>>>>>> 
>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan
<
>>>>>>>> dsetrakyan@apache.org <javascript:;>
>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Is everyone OK with this approach? Should I file
a ticket
>> on
>>>>> it?
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan
<
>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Igniters,
>>>>>>>>>>>> 
>>>>>>>>>>>> There has been lots of talk of proposals
about various
>>> usage
>>>>>>> metrics
>>>>>>>>> for
>>>>>>>>>>>> Ignite and nothing came of it. I would like
to resurrect
>>> the
>>>>>> topic
>>>>>>>> and
>>>>>>>>>>>> propose something very simple and non-intrusive.
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. Update Checker
>>>>>>>>>>>> The main purpose of the update checker is
not to collect
>>>>>> metrics,
>>>>>>>> but
>>>>>>>>> to
>>>>>>>>>>>> notify users about a new version of Ignite
by accessing
>>>>>> maven.org
>>>>>>>> and
>>>>>>>>>>>> getting the version out of the metadata file:
>>>>>>>>>>>> http://repo2.maven.org/maven2/
>>> org/apache/ignite/ignite-core/
>>>>>>>>>>>> maven-metadata.xml
>>>>>>>>>>>> 
>>>>>>>>>>>> This way we do not send any information anywhere
and, at
>>> the
>>>>>> same
>>>>>>>>> time,
>>>>>>>>>>>> urge our users to download and start using
the latest
>>>>> version of
>>>>>>>>> Ignite.
>>>>>>>>>>>> 
>>>>>>>>>>>> 2. Startup Counter
>>>>>>>>>>>> This piece is optional, but we can also get
an insight in
>>> how
>>>>>> many
>>>>>>>>> times
>>>>>>>>>> a
>>>>>>>>>>>> certain Ignite release gets started. This
is just a cool
>>>>> metric
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>> community to gauge the project popularity.
You can think
>> of
>>>>> it
>>>>>> as
>>>>>>>> of a
>>>>>>>>>> page
>>>>>>>>>>>> visit counter shown on many websites. We
can even decide
>> to
>>>>>>> display
>>>>>>>>> this
>>>>>>>>>>>> counter on the Ignite website as well.
>>>>>>>>>>>> 
>>>>>>>>>>>> To do this, we can simply add a JAR in maven
for every
>>>>> release,
>>>>>>> e.g.
>>>>>>>>>>>> ignite-start-counter.jar, which will contain
only 1 byte.
>>>>> Every
>>>>>>> time
>>>>>>>>> an
>>>>>>>>>>>> Ignite node starts, it will download this
JAR in the
>>>>> background.
>>>>>>>> Then
>>>>>>>>> we
>>>>>>>>>>>> will be able to view the number of the total
downloads
>> for
>>>>> this
>>>>>>> JAR
>>>>>>>> in
>>>>>>>>>>>> Maven Central, which is essentially the number
of starts
>> of
>>>>>> Ignite
>>>>>>>>> nodes.
>>>>>>>>>>>> 
>>>>>>>>>>>> *Note that neither of the above suggestions
require
>> Ignite
>>> to
>>>>>> send
>>>>>>>> or
>>>>>>>>>>>> track any user information whatsoever.*
>>>>>>>>>>>> 
>>>>>>>>>>>> Please reply suggesting weather you are OK
with this
>>>>> approach.
>>>>>>>>>>>> 
>>>>>>>>>>>> D.
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message