ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: [DISCUSS] Ignite Update Checker
Date Wed, 25 Oct 2017 06:57:36 GMT
Guys,

I had a quick chat with Nikita earlier today and it seems that we have came across of a blocker
of some kind. With my member's hat on I would love to help to find a resolution that let us
(and potentially other communities) to move forward.

Let's craft the message and circulate it with Infra and Greg. Denis, could you find me tomorrow
at the IMCS and give me an insight into the technical side of it? I feel a bit at loss after
going through this thread...
--
Regards,
  Cos

On October 2, 2017 7:31:48 AM PDT, Denis Magda <dmagda@apache.org> wrote:
>Dmitriy, 
>
>That’s the rule.  See replies in the ticket [1]
>
>*Background: the TLP server is already pretty darned busy just serving
>*static* sites. Dynamic operation for near-200 PMCs would bury the
>machine. Our policy of "static-only websites" has been in place since
>the Foundation started*
>
>Download scripts seem to be the only exception and we as PMC don’t even
>have access to them.
>
>If you want to keep pushing this direction let’s craft a message to
>Greg and Daniel directly. I don’t know what else to ask for here rather
>than a virtual machine that’s conceivably to much for a single script
>like that.
>
>[1] https://issues.apache.org/jira/browse/INFRA-15182
><https://issues.apache.org/jira/browse/INFRA-15182>
>
>—
>Denis
>
>> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <dsetrakyan@apache.org>
>wrote:
>> 
>> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov
><vozerov@gridgain.com>
>> wrote:
>> 
>>> I am not sure it is good idea to send requests to 3rd-party
>addresses from
>>> Ignite node. Let's do not make the same mistakes again.
>>> 
>> 
>> Agree with Vladimir.
>> 
>> We obviously have CGI support on the website. Can someone explain why
>CGI
>> is not possible to use?
>> 
>> 
>>> 
>>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov
><anovikov@gridgain.com>
>>> wrote:
>>> 
>>>> We may directly send request to GA from Ignite node:
>>>> https://developers.google.com/analytics/devguides/
>>> collection/protocol/v1/
>>>> <https://developers.google.com/analytics/devguides/
>>> collection/protocol/v1/
>>>>> 
>>>> Latest version can be received from maven central:
>>>> https://repo1.maven.org/maven2/org/apache/ignite/
>>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
>>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>>>> 
>>>> 
>>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrakyan@apache.org>
>>>> написал(а):
>>>>> 
>>>>> Denis,
>>>>> 
>>>>> I am not sure I understand. We already do have CGI enabled for
>>>>> download.cgi. Is there something else we need?
>>>>> 
>>>>> D.
>>>>> 
>>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dmagda@gridgain.com>
>>> wrote:
>>>>> 
>>>>>> There is an obstacle. There is no way to execute the script using
>PHP
>>> or
>>>>>> similar sever side language and trigger GA as discussed earlier:
>>>>>> https://issues.apache.org/jira/browse/INFRA-15182
>>>>>> 
>>>>>> How else can we tackle this?
>>>>>> 
>>>>>> Denis
>>>>>> 
>>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
>>>> dsetrakyan@apache.org>
>>>>>> wrote:
>>>>>> 
>>>>>>> I think it is safe to assume at this point that everyone is in
>>> general
>>>>>>> agreement, since there are no active objections.
>>>>>>> 
>>>>>>> I have filed a ticket for the 2.3 release. Let's try to make
it
>>> happen:
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
>>>>>>> 
>>>>>>> D.
>>>>>>> 
>>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>>>>>> dsetrakyan@apache.org
>>>>>>> <javascript:;>>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>>> raul.asf@evosent.com
>>>>>>> <javascript:;>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Yeah, I guess that's doable as well and requires less
>management
>>>>>> effort
>>>>>>>>> than my suggestion. We could use events [1] to store
payload
>data
>>>>>> (e.g.
>>>>>>>>> IP,
>>>>>>>>> version, etc.)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Yes, we could use events or some other similar API provided
by
>GA.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> What the download page CGI developed in? PHP?
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> To be honest, no clue. I guess someone in the community can
>figure
>>> it
>>>>>>> out:
>>>>>>>>
>https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> However, I'm not sure whether storing this data in a
3rd party
>>>>>> (Google)
>>>>>>> is
>>>>>>>>> compliant with the ASF policy. I guess it's no biggie,
but if
>>> there's
>>>>>>>>> doubt
>>>>>>>>> in the PMC, it's better to ask legal@.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I am not sure there is anything to ask about. The whole Ignite
>>> website
>>>>>> is
>>>>>>>> GA enabled, and all we are doing is accessing a standard
web
>page
>>> from
>>>>>>> the
>>>>>>>> Ignite web site. The information gathered from GA is available
>only
>>> to
>>>>>>> the
>>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused
by
>this
>>>>>>>> question.
>>>>>>>> 
>>>>>>>> Even ASF website itself uses GA: https://www.apache.org/
>>>>>>>> foundation/policies/privacy.html
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> Sure, would be nice to hear from Roman as well.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Cheers.
>>>>>>>>> 
>>>>>>>>> [1]
>>>>>>>>> https://developers.google.com/analytics/devguides/collection
>>>>>>>>> /analyticsjs/events
>>>>>>>>> 
>>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Raul,
>>>>>>>>>> 
>>>>>>>>>> Could point about Javascript, it will not work because
it
>executes
>>>>>> in
>>>>>>>>> the
>>>>>>>>>> browser. This means we need a server-side script,
like CGI we
>are
>>>>>>> using
>>>>>>>>> on
>>>>>>>>>> our download page.
>>>>>>>>>> 
>>>>>>>>>> How about this approach. We create something like
>>> ignite-version.cgi
>>>>>>>>> script
>>>>>>>>>> which will invoke a call to GA and then return the
latest
>version.
>>>>>>>>>> 
>>>>>>>>>> This should work, right?
>>>>>>>>>> 
>>>>>>>>>> D.
>>>>>>>>>> 
>>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani
<
>>>>>> raul.asf@evosent.com
>>>>>>> <javascript:;>>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey Dmitriy and all
>>>>>>>>>>> 
>>>>>>>>>>> Also, since we have GA enabled for the website,
we can track
>how
>>>>>>> many
>>>>>>>>>> times
>>>>>>>>>>>> this page was accessed, which will be equal
to the number
>of
>>>>>>> starts.
>>>>>>>>>> This
>>>>>>>>>>>> way, the counter information is tracked and
monitored by
>the
>>>>>>> Ignite
>>>>>>>>>> PMC.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Unfortunately this won't work because GA is loaded
via
>Javascript
>>>>>> on
>>>>>>>>> the
>>>>>>>>>>> browser, so Google will never receive the page
hit.
>>>>>>>>>>> 
>>>>>>>>>>> Given the constraints, the most viable solution
is an HTTPS
>>>>>> endpoint
>>>>>>>>>>> running on ASF infra that Ignite invokes via
a GET or POST
>>>>>> request.
>>>>>>>>> The
>>>>>>>>>>> simplest thing is to write each request in a
log file, along
>with
>>>>>>> the
>>>>>>>>>>> timestamp, the version reported by the client,
maybe the IP
>(not
>>>>>>> sure
>>>>>>>>>> about
>>>>>>>>>>> the ASF rules about this concerning privacy,
I guess it's OK
>if
>>>>>> you
>>>>>>>>> make
>>>>>>>>>> it
>>>>>>>>>>> an opt-in) and a unique node identifier, i.e.
a UUID the
>node
>>>>>>> creates
>>>>>>>>> on
>>>>>>>>>>> first startup or something.
>>>>>>>>>>> 
>>>>>>>>>>> This endpoint would need some basic DDoS protection
and
>>>>>> blacklisting
>>>>>>>>> to
>>>>>>>>>>> prevent data spoofing.
>>>>>>>>>>> 
>>>>>>>>>>> If we'll be implementing this endpoint anyway,
then there's
>no
>>>>>> point
>>>>>>>>>>> placing another file on Git or elsewhere for
reporting the
>latest
>>>>>>>>>> versions:
>>>>>>>>>>> the endpoint itself can return them.
>>>>>>>>>>> 
>>>>>>>>>>> WDYT?
>>>>>>>>>>> 
>>>>>>>>>>> Cheers.
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan
<
>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Cos, Raul,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the feedback. I completely agree
about Maven
>Central
>>>>>>>>> being a
>>>>>>>>>>> 3rd
>>>>>>>>>>>> party repo (did not think about that initially).
All your
>>>>>>>>> suggestions
>>>>>>>>>>> make
>>>>>>>>>>>> sense, but I would like to keep it as simple
as possible,
>and so
>>>>>>> far
>>>>>>>>>>>> everything suggested required GIT dependencies
and extra
>work.
>>>>>>>>>>>> 
>>>>>>>>>>>> How about Yakov's suggestion. We simply add
a page to the
>Ignite
>>>>>>>>>> website
>>>>>>>>>>>> which will have only the latest version.
Every time a node
>>>>>> starts,
>>>>>>>>> it
>>>>>>>>>>>> receives the latest version from the page
and suggests that
>>>>>> users
>>>>>>>>>> upgrade
>>>>>>>>>>>> if needed.
>>>>>>>>>>>> 
>>>>>>>>>>>> Also, since we have GA enabled for the website,
we can
>track how
>>>>>>>>> many
>>>>>>>>>>> times
>>>>>>>>>>>> this page was accessed, which will be equal
to the number
>of
>>>>>>> starts.
>>>>>>>>>> This
>>>>>>>>>>>> way, the counter information is tracked and
monitored by
>the
>>>>>>> Ignite
>>>>>>>>>> PMC.
>>>>>>>>>>>> 
>>>>>>>>>>>> This approach looks pretty innocent to me
and everything is
>kept
>>>>>>> and
>>>>>>>>>>>> managed within Apache.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>> 
>>>>>>>>>>>> D.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin
Boudnik <
>>>>>>>>> cos@apache.org <javascript:;>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> I agree with Raul.
>>>>>>>>>>>>> - providing a ping-back address to a
3rd party might be
>frown
>>>>>>>>> upon by
>>>>>>>>>>>> some.
>>>>>>>>>>>>> And might have a consequences like stats
collection about
>>>>>>> users'
>>>>>>>>>>>>> infrastructure.
>>>>>>>>>>>>> - checking an ASF git-repo is easy and
won't download any
>>>>>> binary
>>>>>>>>>> data:
>>>>>>>>>>>>> everything is clear text and could be
easily monitored by
>>>>>> any
>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>> network
>>>>>>>>>>>>> diagnostic tools, shall it be required.
But it involves a
>>>>>> bit
>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>> release
>>>>>>>>>>>>> discipline.
>>>>>>>>>>>>> - the binary data download in the runtime
is my main
>concern.
>>>>>>>>> This is
>>>>>>>>>>> the
>>>>>>>>>>>>> vector of MMA. What if someone gains
the control over the
>>>>>>>>>> repository
>>>>>>>>>>>> and
>>>>>>>>>>>>> replaces the file with some malicious
content.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As for the particular mechanism: IIRC
Ignite used to make
>a
>>>>>> call
>>>>>>>>> to
>>>>>>>>>> an
>>>>>>>>>>>>> external script to check upon the atest
software version
>>>>>>> available
>>>>>>>>>> for
>>>>>>>>>>>>> download. In the past, the endpoint was
running on a 3rd
>party
>>>>>>>>>> server,
>>>>>>>>>>> I
>>>>>>>>>>>>> believe the best way would be to put
this script on ASF
>infra
>>>>>>> and
>>>>>>>>>> have
>>>>>>>>>>>> the
>>>>>>>>>>>>> "update checker" running in a completely
controlled
>>>>>> environment.
>>>>>>>>>>>> Actually,
>>>>>>>>>>>>> I
>>>>>>>>>>>>> recall we had this very discussion during
the Incubation;
>I
>>>>>> can
>>>>>>>>>>> probably
>>>>>>>>>>>>> dig
>>>>>>>>>>>>> out the corresponding thread.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>> Cok
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul
Kripalani wrote:
>>>>>>>>>>>>>> Hey guys
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In my opinion, maven.org is still
owned by a third party
>>>>>>>>>> (Sonatype).
>>>>>>>>>>>> We
>>>>>>>>>>>>>> don't know what kind of data analysis
or intelligence
>>>>>>> extraction
>>>>>>>>>> they
>>>>>>>>>>>>> run.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If Ignite servers all over the world
were hitting
>maven.org
>>>>>>>>>>>> periodically
>>>>>>>>>>>>>> asking for an Ignite Maven artifact,
it gives Sonatype a
>>>>>> clear
>>>>>>>>>>>> indication
>>>>>>>>>>>>>> of who is running an Ignite server.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> They could reverse lookup the IP
address and find out
>what
>>>>>>>>>>> corporation
>>>>>>>>>>>> it
>>>>>>>>>>>>>> is.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> How about having Ignite check the
ASF Git directly?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We could use the Git ssh API, but
that would require a
>new
>>>>>>>>>>> dependency,
>>>>>>>>>>>>>> which I advise against.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Alternatively, we could have it scrape
this HTML for new
>Git
>>>>>>>>> tags:
>>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Another option is to place a txt
file in the root of the
>>>>>>> master
>>>>>>>>>>> branch
>>>>>>>>>>>>> (e.g
>>>>>>>>>>>>>> LATEST), containing a list of the
latest GA versions for
>>>>>> each
>>>>>>>>> major
>>>>>>>>>>>>> version
>>>>>>>>>>>>>> line (1.x, 2.x).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would advocate this last option,
but it requires
>somebody
>>>>>>>>>>> remembering
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> update the file with every release,
unless we automate it
>>>>>>> with a
>>>>>>>>>>> Maven
>>>>>>>>>>>>>> plugin.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hope that helps!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda"
<dmagda@apache.org
>>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I see nothing wrong with this approach.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans,
what do you think?
>Is
>>>>>> it
>>>>>>>>> good
>>>>>>>>>>> to
>>>>>>>>>>>>> go?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> —
>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM,
Dmitriy Setrakyan <
>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Is everyone OK with this approach?
Should I file a
>ticket
>>>>>> on
>>>>>>>>> it?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07
PM, Dmitriy Setrakyan <
>>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> There has been lots of talk
of proposals about various
>>>>>>> usage
>>>>>>>>>>> metrics
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> Ignite and nothing came of
it. I would like to
>resurrect
>>>>>>> the
>>>>>>>>>> topic
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> propose something very simple
and non-intrusive.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1. Update Checker
>>>>>>>>>>>>>>>> The main purpose of the update
checker is not to
>collect
>>>>>>>>>> metrics,
>>>>>>>>>>>> but
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> notify users about a new
version of Ignite by accessing
>>>>>>>>>> maven.org
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> getting the version out of
the metadata file:
>>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
>>>>>>> org/apache/ignite/ignite-core/
>>>>>>>>>>>>>>>> maven-metadata.xml
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This way we do not send any
information anywhere and,
>at
>>>>>>> the
>>>>>>>>>> same
>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>> urge our users to download
and start using the latest
>>>>>>>>> version of
>>>>>>>>>>>>> Ignite.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2. Startup Counter
>>>>>>>>>>>>>>>> This piece is optional, but
we can also get an insight
>in
>>>>>>> how
>>>>>>>>>> many
>>>>>>>>>>>>> times
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> certain Ignite release gets
started. This is just a
>cool
>>>>>>>>> metric
>>>>>>>>>>> for
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> community to gauge the project
popularity. You can
>think
>>>>>> of
>>>>>>>>> it
>>>>>>>>>> as
>>>>>>>>>>>> of a
>>>>>>>>>>>>>> page
>>>>>>>>>>>>>>>> visit counter shown on many
websites. We can even
>decide
>>>>>> to
>>>>>>>>>>> display
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> counter on the Ignite website
as well.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> To do this, we can simply
add a JAR in maven for every
>>>>>>>>> release,
>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>> ignite-start-counter.jar,
which will contain only 1
>byte.
>>>>>>>>> Every
>>>>>>>>>>> time
>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>> Ignite node starts, it will
download this JAR in the
>>>>>>>>> background.
>>>>>>>>>>>> Then
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> will be able to view the
number of the total downloads
>>>>>> for
>>>>>>>>> this
>>>>>>>>>>> JAR
>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> Maven Central, which is essentially
the number of
>starts
>>>>>> of
>>>>>>>>>> Ignite
>>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> *Note that neither of the
above suggestions require
>>>>>> Ignite
>>>>>>> to
>>>>>>>>>> send
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> track any user information
whatsoever.*
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Please reply suggesting weather
you are OK with this
>>>>>>>>> approach.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 

Mime
View raw message