ofbiz-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Giulio Speri (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (OFBIZ-10957) Visit/Visitor specific client IPs tracking exclusion
Date Thu, 25 Apr 2019 09:27:00 GMT

     [ https://issues.apache.org/jira/browse/OFBIZ-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Giulio Speri updated OFBIZ-10957:
    Labels: ip-exclusion tracking visit  (was: )

> Visit/Visitor specific client IPs tracking exclusion
> ----------------------------------------------------
>                 Key: OFBIZ-10957
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-10957
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: framework
>    Affects Versions: Release Branch 13.07
>            Reporter: Giulio Speri
>            Assignee: Giulio Speri
>            Priority: Major
>              Labels: ip-exclusion, tracking, visit
> I shortly explain the server architecture on which OFBiz is running: hosted by a third
party supplier, there are two (virtual) machines where Apache OFBiz 13.07.03 is running behind
Apache2 web server (so we have two web fronts).
> On other two different machines there are the database (MariaDB) and HaProxy has a load
> HaProxy is configured to perform its Health Checks on the backend servers with a Http
GET on the Home Page of one of the two sites.
> Visit and Visitor tracking are enabled, for BI and analytics purposes, so we cannot turn
them off.
> These two combined things caused the Visit and Visitor tables to explode in dimensions
(we counted about 19M records of Visit and about 67M of Visitors, with the 86% of those caused
by the load balancer), since each hit of the HaProxy store a Visit and a Visitor record on
the db (plus some other record of other entities, like ShoppingList, due to <firstvisit>
and <preprocessor> events).
> A bad side effect of this situation, on the long run, is an overall performance degradation,
and an increase in webfront unavailability time windows during the day: it's not necessary
to say that our customer was not so happy about this.
> The difficult part of figuring out this problem, was that we did not have direct access
to HaProxy and DB machines, to check logs. 
> The solution we thought and implemented, was to exclude from Visit/Visitor tracking specific
IP addresses (for our case we were interested in HaProxy IP).
> The Visit and Visitor records (along with firstvisit and preprocessor events) are created
mainly in the ControlServlet class, using VisitHandler getVisit/getVisitor/getVisitId methods.
> Our idea consist in reading from a .properties file one or more IP addresses we would
like to exclude from tracking and then check them against the client ip address the request
is coming from.
> If the client ip address is in the "exclusion list", then do not persist visit/visitor
and do not run firstvisit events neither for it.
> The idea is quite simple, but we noticed in few days, a meaningful improvement in overall
system performance and stability/availability.
> This kind of exclusion could be also useful in case we do not want to track or register
internal IP addresses (ie: mainly used for testing).
> However this solution, should be integrated with a service (cron or scheduled in ofbiz)
that keeps the number of records in the tables limited (for example keep only the last month
of visit/visitor); I think that these two solutions together, could do the job well.

This message was sent by Atlassian JIRA

View raw message