cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olivier Lemasle (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-5404) Network usages (bytes sent/received) are saved in the wrong timezone
Date Fri, 06 Dec 2013 17:01:37 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-5404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Olivier Lemasle updated CLOUDSTACK-5404:
----------------------------------------

    Description: 
On ACS 4.2, usages with types 4 and 5 (Network bytes sent / Network bytes received) are not
stored in database (table cloud_usage.cloud_usage) with the same timezone than the other usages.

On my CloudStack 4.2 installation (using GMT+1), network usages appear in database one hour
later than the other usage types:

mysql> select usage_type, max(end_date) from cloud_usage group by usage_type;
+------------+---------------------+
| usage_type | max(end_date)       |
+------------+---------------------+
|          1 | 2013-12-06 14:59:59 |
|          2 | 2013-12-06 14:59:59 |
|          3 | 2013-12-06 14:59:59 |
|          4 | 2013-12-06 15:59:59 |
|          5 | 2013-12-06 15:59:59 |
|          6 | 2013-12-06 14:59:59 |
|         13 | 2013-12-06 14:59:59 |
+------------+---------------------+
7 rows in set (0.00 sec)

In fact:
- for network bytes sent/received, usages are stored in local timezone (in my case GMT+1)
- for every other usage types, usages are stored in GMT.

I checked the SQL requests and here are two consecutive requests:

INSERT INTO cloud_usage.cloud_usage (zone_id, account_id, domain_id, description, usage_display,
usage_type, raw_usage, vm_instance_id, vm_name, offering_id, template_id, usage_id, type,
size, network_id, start_date, end_date, virtual_size) VALUES (1,2,1,'network bytes received
for Host: 4','0 bytes received',5,0.0,null,null, null, null, 4,'DomainRouter',null,204,'2013-12-06
15:00:00','2013-12-06 15:59:59',null)

INSERT INTO cloud_usage (cloud_usage.zone_id, cloud_usage.account_id, cloud_usage.domain_id,
cloud_usage.description, cloud_usage.usage_display, cloud_usage.usage_type, cloud_usage.raw_usage,
cloud_usage.vm_instance_id, cloud_usage.vm_name, cloud_usage.offering_id, cloud_usage.template_id,
cloud_usage.usage_id, cloud_usage.type, cloud_usage.size, cloud_usage.virtual_size, cloud_usage.network_id,
cloud_usage.start_date, cloud_usage.end_date) VALUES (1, 2, 1, _binary'Volume Id: 3 usage
time', _binary'1 Hrs', 6, 1.0, null, null, null, null, 3, null, 21474836480, null, null, '2013-12-06
14:00:00', '2013-12-06 14:59:59')

The first, for Network bytes sent, is a request made in com.cloud.usage.dao.UsageDaoImpl.saveUsageRecords(List<UsageVO>),
and the second is done with com.cloud.utils.db.GenericDaoBase.persist(T).

The issue comes from the fix for CLOUDSTACK-2707, where, for performance reasons, multiple
calls to persist() in com.cloud.usage.parser.NetworkUsageParser were replaced by a single
call to saveUsageRecords() to process the insertions as a batch.

So, NetworkUsageParser uses this saveUsageRecords() function with no special timezone management:

pstmt.setTimestamp(16, new Timestamp(usageRecord.getStartDate().getTime()));
pstmt.setTimestamp(17, new Timestamp(usageRecord.getEndDate().getTime()));

whereas for other usages, parsers (VMInstanceUsageParser, VolumeUsageParser, NetworkOfferingUsageParser,...)
use persist() which has a special timezone management converting every date in GMT:

else if (attr.field.getType() == Date.class) {
  final Date date = (Date)value;
  if (date == null) {
      pstmt.setObject(j, null);
      return;
  }
  if (attr.is(Attribute.Flag.Date)) {
      pstmt.setString(j, DateUtil.getDateDisplayString(s_gmtTimeZone, date));
  } else if (attr.is(Attribute.Flag.TimeStamp)) {
      pstmt.setString(j, DateUtil.getDateDisplayString(s_gmtTimeZone, date));
  } else if (attr.is(Attribute.Flag.Time)) {
      pstmt.setString(j, DateUtil.getDateDisplayString(s_gmtTimeZone, date));
  }
}

  was:
On ACS 4.2, usages with types 4 and 5 (Network bytes sent / Network bytes received) are not
stored in database (table cloud_usage.cloud_usage) with the same timezone than the other usages.

On my CloudStack 4.2 installation (using GMT+1), network usages appear in database one hour
later than the other usage types:

{noformat}
mysql> select usage_type, max(end_date) from cloud_usage group by usage_type;
+------------+---------------------+
| usage_type | max(end_date)       |
+------------+---------------------+
|          1 | 2013-12-06 14:59:59 |
|          2 | 2013-12-06 14:59:59 |
|          3 | 2013-12-06 14:59:59 |
|          4 | 2013-12-06 15:59:59 |
|          5 | 2013-12-06 15:59:59 |
|          6 | 2013-12-06 14:59:59 |
|         13 | 2013-12-06 14:59:59 |
+------------+---------------------+
7 rows in set (0.00 sec)
{noformat}

In fact:
- for network bytes sent/received, usages are stored in local timezone (in my case GMT+1)
- for every other usage types, usages are stored in GMT.

I checked the SQL requests and here are two consecutive requests:

{code:sql}
INSERT INTO cloud_usage.cloud_usage (zone_id, account_id, domain_id, description, usage_display,
usage_type, raw_usage, vm_instance_id, vm_name, offering_id, template_id, usage_id, type,
size, network_id, start_date, end_date, virtual_size) VALUES (1,2,1,'network bytes received
for Host: 4','0 bytes received',5,0.0,null,null, null, null, 4,'DomainRouter',null,204,'2013-12-06
15:00:00','2013-12-06 15:59:59',null)

INSERT INTO cloud_usage (cloud_usage.zone_id, cloud_usage.account_id, cloud_usage.domain_id,
cloud_usage.description, cloud_usage.usage_display, cloud_usage.usage_type, cloud_usage.raw_usage,
cloud_usage.vm_instance_id, cloud_usage.vm_name, cloud_usage.offering_id, cloud_usage.template_id,
cloud_usage.usage_id, cloud_usage.type, cloud_usage.size, cloud_usage.virtual_size, cloud_usage.network_id,
cloud_usage.start_date, cloud_usage.end_date) VALUES (1, 2, 1, _binary'Volume Id: 3 usage
time', _binary'1 Hrs', 6, 1.0, null, null, null, null, 3, null, 21474836480, null, null, '2013-12-06
14:00:00', '2013-12-06 14:59:59')
{code}

The first, for Network bytes sent, is a request made in com.cloud.usage.dao.UsageDaoImpl.saveUsageRecords(List<UsageVO>),
and the second is done with com.cloud.utils.db.GenericDaoBase.persist(T).

The issue comes from the fix for CLOUDSTACK-2707, where, for performance reasons, multiple
calls to persist() in com.cloud.usage.parser.NetworkUsageParser were replaced by a single
call to saveUsageRecords() to process the insertions as a batch.

So, NetworkUsageParser uses this saveUsageRecords() function with no special timezone management:

{code:java}
pstmt.setTimestamp(16, new Timestamp(usageRecord.getStartDate().getTime()));
pstmt.setTimestamp(17, new Timestamp(usageRecord.getEndDate().getTime()));
{code}

whereas for other usages, parsers (VMInstanceUsageParser, VolumeUsageParser, NetworkOfferingUsageParser,...)
use persist() which has a special timezone management converting every date in GMT:

{code:java}
else if (attr.field.getType() == Date.class) {
  final Date date = (Date)value;
  if (date == null) {
      pstmt.setObject(j, null);
      return;
  }
  if (attr.is(Attribute.Flag.Date)) {
      pstmt.setString(j, DateUtil.getDateDisplayString(s_gmtTimeZone, date));
  } else if (attr.is(Attribute.Flag.TimeStamp)) {
      pstmt.setString(j, DateUtil.getDateDisplayString(s_gmtTimeZone, date));
  } else if (attr.is(Attribute.Flag.Time)) {
      pstmt.setString(j, DateUtil.getDateDisplayString(s_gmtTimeZone, date));
  }
}
{code}


> Network usages (bytes sent/received) are saved in the wrong timezone
> --------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5404
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5404
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Usage
>    Affects Versions: 4.2.0
>            Reporter: Olivier Lemasle
>
> On ACS 4.2, usages with types 4 and 5 (Network bytes sent / Network bytes received) are
not stored in database (table cloud_usage.cloud_usage) with the same timezone than the other
usages.
> On my CloudStack 4.2 installation (using GMT+1), network usages appear in database one
hour later than the other usage types:
> mysql> select usage_type, max(end_date) from cloud_usage group by usage_type;
> +------------+---------------------+
> | usage_type | max(end_date)       |
> +------------+---------------------+
> |          1 | 2013-12-06 14:59:59 |
> |          2 | 2013-12-06 14:59:59 |
> |          3 | 2013-12-06 14:59:59 |
> |          4 | 2013-12-06 15:59:59 |
> |          5 | 2013-12-06 15:59:59 |
> |          6 | 2013-12-06 14:59:59 |
> |         13 | 2013-12-06 14:59:59 |
> +------------+---------------------+
> 7 rows in set (0.00 sec)
> In fact:
> - for network bytes sent/received, usages are stored in local timezone (in my case GMT+1)
> - for every other usage types, usages are stored in GMT.
> I checked the SQL requests and here are two consecutive requests:
> INSERT INTO cloud_usage.cloud_usage (zone_id, account_id, domain_id, description, usage_display,
usage_type, raw_usage, vm_instance_id, vm_name, offering_id, template_id, usage_id, type,
size, network_id, start_date, end_date, virtual_size) VALUES (1,2,1,'network bytes received
for Host: 4','0 bytes received',5,0.0,null,null, null, null, 4,'DomainRouter',null,204,'2013-12-06
15:00:00','2013-12-06 15:59:59',null)
> INSERT INTO cloud_usage (cloud_usage.zone_id, cloud_usage.account_id, cloud_usage.domain_id,
cloud_usage.description, cloud_usage.usage_display, cloud_usage.usage_type, cloud_usage.raw_usage,
cloud_usage.vm_instance_id, cloud_usage.vm_name, cloud_usage.offering_id, cloud_usage.template_id,
cloud_usage.usage_id, cloud_usage.type, cloud_usage.size, cloud_usage.virtual_size, cloud_usage.network_id,
cloud_usage.start_date, cloud_usage.end_date) VALUES (1, 2, 1, _binary'Volume Id: 3 usage
time', _binary'1 Hrs', 6, 1.0, null, null, null, null, 3, null, 21474836480, null, null, '2013-12-06
14:00:00', '2013-12-06 14:59:59')
> The first, for Network bytes sent, is a request made in com.cloud.usage.dao.UsageDaoImpl.saveUsageRecords(List<UsageVO>),
and the second is done with com.cloud.utils.db.GenericDaoBase.persist(T).
> The issue comes from the fix for CLOUDSTACK-2707, where, for performance reasons, multiple
calls to persist() in com.cloud.usage.parser.NetworkUsageParser were replaced by a single
call to saveUsageRecords() to process the insertions as a batch.
> So, NetworkUsageParser uses this saveUsageRecords() function with no special timezone
management:
> pstmt.setTimestamp(16, new Timestamp(usageRecord.getStartDate().getTime()));
> pstmt.setTimestamp(17, new Timestamp(usageRecord.getEndDate().getTime()));
> whereas for other usages, parsers (VMInstanceUsageParser, VolumeUsageParser, NetworkOfferingUsageParser,...)
use persist() which has a special timezone management converting every date in GMT:
> else if (attr.field.getType() == Date.class) {
>   final Date date = (Date)value;
>   if (date == null) {
>       pstmt.setObject(j, null);
>       return;
>   }
>   if (attr.is(Attribute.Flag.Date)) {
>       pstmt.setString(j, DateUtil.getDateDisplayString(s_gmtTimeZone, date));
>   } else if (attr.is(Attribute.Flag.TimeStamp)) {
>       pstmt.setString(j, DateUtil.getDateDisplayString(s_gmtTimeZone, date));
>   } else if (attr.is(Attribute.Flag.Time)) {
>       pstmt.setString(j, DateUtil.getDateDisplayString(s_gmtTimeZone, date));
>   }
> }



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message