kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neil Avery (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-5515) Consider removing date formatting from Segments class
Date Wed, 05 Jul 2017 11:04:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068546#comment-16068546
] 

Neil Avery edited comment on KAFKA-5515 at 7/5/17 11:03 AM:
------------------------------------------------------------

*Investigation:*
Taking a look at the use shows SimpleDateFormat (SFD) is used for *parsing* Segment file names
during initialisation, and *format* ting during runtime. I presume the suggested problem lies
in the formatting.

*Micro benchmark SDF*
Formatting 1,000,000 items takes 250ms once hotspotting has kicked in.  Per/M items (ms):
[707, 572, 543, 591, 546.0, 545.0, 363.0, 250 etc]
Parsing is slow - 2500ms per 1,000,000 items

Commons-lang3-FastDateFormat is available in the project but not as a dependency on this particular
module. FDF micro-bench starts at 400ms/million then gets down to 350ms (not very convincing).


Calendar usage sucks performance and there is a degree of caching inside both of the impls.


Looking at this in a different way "Segments" is a time-series slice/bucketing function to
group/allocate/lookup segments etc. 

I've knocked together a simple math alternative that breaks into time-slice where all months/years
are equals size i.e. not using a calendar - you get an approximate idea of performance: 150-200ms
without hotspotting. The problem is that a real-calendar is still used upon initialisation
extract segment-ids - there will be inconsistencies and likely breakage.

*Best performance*
The best alternative would be to ditch calendars for parsing and formatting and to trunc/floor
unix time to minutes/hours etc (at the cost a segment-filename readability). I'm not sure
if there will be operational upgrade paths etc in order to make the change seamless. 


was (Author: neil.avery):
*Investigation:*
Taking a look at the use shows SimpleDateFormat (SFD) is used for *parsing* Segment file names
during initialisation, and *format*ting during runtime. I presume the suggested problem lies
in the formatting.

*Micro benchmark SDF*
Formatting 1,000,000 items takes 250ms once hotspotting has kicked in.  Per/M items (ms):
[707, 572, 543, 591, 546.0, 545.0, 363.0, 250 etc]
Parsing is slow - 2500ms per 1,000,000 items

Commons-lang3-FastDateFormat is available in the project but not as a dependency on this particular
module. FDF micro-bench starts at 400ms/million then gets down to 350ms (not very convincing).


Calendar usage sucks performance and there is a degree of caching inside both of the impls.


Looking at this in a different way "Segments" is a time-series slice/bucketing function to
group/allocate/lookup segments etc. 

I've knocked together a simple math alternative that breaks into time-slice where all months/years
are equals size i.e. not using a calendar - you get an approximate idea of performance: 150-200ms
without hotspotting. The problem is that a real-calendar is still used upon initialisation
extract segment-ids - there will be inconsistencies and likely breakage.

*Best performance*
The best alternative would be to ditch calendars for parsing and formatting and to trunc/floor
unix time to minutes/hours etc (at the cost a segment-filename readability). I'm not sure
if there will be operational upgrade paths etc in order to make the change seamless. 

> Consider removing date formatting from Segments class
> -----------------------------------------------------
>
>                 Key: KAFKA-5515
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5515
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Bill Bejeck
>            Assignee: Neil Avery
>              Labels: performance
>
> Currently the {{Segments}} class uses a date when calculating the segment id and uses
{{SimpleDateFormat}} for formatting the segment id.  However this is a high volume code path
and creating a new {{SimpleDateFormat}} and formatting each segment id is expensive.  We should
look into removing the date from the segment id or at a minimum use a faster alternative to
{{SimpleDateFormat}}.  We should also consider keeping a lookup of existing segments to avoid
as many string operations as possible.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message