incubator-ooo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Weir <robw...@apache.org>
Subject Re: Download stats script (in progress)
Date Sun, 13 May 2012 15:17:46 GMT
On Fri, May 11, 2012 at 2:58 PM, Roberto Galoppini <rgaloppini@geek.net> wrote:
> On Fri, May 11, 2012 at 5:06 AM, Rob Weir <robweir@apache.org> wrote:
>
>> SourceForge has a nice REST API to query for download stats and return
>> them in JSON objections.  Unfortunately, our directory structure for
>> AOO 3.4 is rather odd, with English downloads in one place,
>> translations in another directory, and mixing hashes, installs and
>> languages packs altogether.  So getting these stats is a little
>> painful.  You can't just get the numbers of a single directory and be
>> done.  It is more complicated than that.
>>
>> Also, the SF API seems to be rate limited, or at least I'm getting
>> errors if I query it too much.  That's understandable.
>>
>
> Rob, can provide me with more info about this, so that we can investigate
> it further?


Hi Roberto,

At the python level the error is on a urllib.urlopen().read(), with an
returned error of:

"IOError: [Errno socket error] [Errno 10054] An existing connection
was forcibly closed by the remote host"

I find that this happens when I make many (> 50) requests in a short
period of time (1 or 2 minutes).  My solution right now is to maintain
a local disk cache of prior returned results.  Not only does this
reduce the number of requests I send SF, but it also improves the
speed of the report generation.

In any case, I'm happy with the caching solution, so this is not a
blocking issue for me right now.

-Rob

> We plan to share some stats figures next week, as we did previously for
> Extensions and Templates.
>
>
> Thanks,
>
> Roberto
>
>
>>
>> So.... I'm coding a simple download stats app, in python, that will
>> collect together all the relevant stats and produce reports.  It
>> caches on disk JSON objects that have already been retrieved, which
>> eliminates the throttling issues as well as greatly improves
>> performance.
>>
>> Not quite done, but I'll check it in (where?) when it is fully
>> debugged and validated.  My goal is to have solid numbers for the one
>> week mark next Tuesday.  And from what I'm seeing so far, the numbers
>> will be amazing.
>>
>> But two quick questions to help me finish this:
>>
>> 1) Historically, what did OOo report as "downloads"?  Was this just a
>> count of full installs?  Or language packs as well?
>>
>> 2) It is easy to produce downloads by language and platform, since our
>> installs are already defined that way.  But I can also report
>> per-country.  Is that interesting to anyone?   For example, in Canada,
>> the most popular downloads are X, Y, Z.
>>
>>
>> -Rob
>>
>
> --
> ====
> This e- mail message is intended only for the named recipient(s) above. It
> may contain confidential and privileged information. If you are not the
> intended recipient you are hereby notified that any dissemination,
> distribution or copying of this e-mail and any attachment(s) is strictly
> prohibited. If you have received this e-mail in error, please immediately
> notify the sender by replying to this e-mail and delete the message and any
> attachment(s) from your system. Thank you.
>

Mime
View raw message