kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharon Lucas (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-6572) kafka-consumer-groups does not reset offsets to specified datetime correctly
Date Mon, 19 Feb 2018 22:04:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharon Lucas updated KAFKA-6572:
--------------------------------
    Description: 
With Kafka 1.0.0 we are seeing a problem using the kafka-consumer-groups.sh --reset-offsets option to reset offsets to a specific date/time in our production environment.

We first tried to use the kafka-consumer-groups.sh command with the --reset-offsets option and with option --to-datetime 2018-02-10T00:00:00.000 in our staging environment and it worked correctly.  Running the following command changed it to start processing logs from February 12, 2018 (4 days ago) for a topic that had a large lag.  We did a dry run to verify before running with the --execute option.
{code:java}
root@ossmlpstagemon0101a:/# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.NNN.NN.NN:9092 --group logstash-elasticsearch-latest --to-datetime 2018-02-12T00:00:00.000-06:00 --reset-offsets --topic staging-mon01-rg-elasticsearch --execute{code}
We stopped the kafka mirrors that process this topic before resetting the offsets and started the kafka mirrors after rsetting the offsets.  We verified that it correctly started processing logs from February 12, 2018.

Then we tried resetting offsets in a production environment for a topic that had a very large lag using option --to-datetime 2018-02-10T00:00:00.000 and it did not work as expected. We stopped the kafka mirrors that process this topic before resetting the offsets and did a dry run to see what the new offsets would be:
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
Note: This will not show information about old Zookeeper-based consumers.

^@^@^@^@
TOPIC                          PARTITION  NEW-OFFSET
prod_in-ams03-geo-ca_access    52         52084147
prod_in-ams03-geo-ca_access    106        52154199
prod_in-ams03-geo-ca_access    75         52148673
prod_in-ams03-geo-ca_access    61         52130753
prod_in-ams03-geo-ca_access    49         52151667
prod_in-ams03-geo-ca_access    48         52145233
prod_in-ams03-geo-ca_access    27         52092805
prod_in-ams03-geo-ca_access    26         52139644
prod_in-ams03-geo-ca_access    65         52157504
prod_in-ams03-geo-ca_access    105        52166289
prod_in-ams03-geo-ca_access    38         52160464
prod_in-ams03-geo-ca_access    22         52093451
prod_in-ams03-geo-ca_access    4          52151660
prod_in-ams03-geo-ca_access    90         52160296
prod_in-ams03-geo-ca_access    25         52161691
prod_in-ams03-geo-ca_access    13         52145828
prod_in-ams03-geo-ca_access    56         52162867
prod_in-ams03-geo-ca_access    42         52072094
prod_in-ams03-geo-ca_access    7          52069496
prod_in-ams03-geo-ca_access    117        52087078
prod_in-ams03-geo-ca_access    32         52073732
prod_in-ams03-geo-ca_access    102        52082022
prod_in-ams03-geo-ca_access    76         52141018
prod_in-ams03-geo-ca_access    83         52154542
prod_in-ams03-geo-ca_access    72         52095051
prod_in-ams03-geo-ca_access    85         52149907
prod_in-ams03-geo-ca_access    119        52134435
prod_in-ams03-geo-ca_access    113        52159340
prod_in-ams03-geo-ca_access    55         52146597
prod_in-ams03-geo-ca_access    18         52149079
prod_in-ams03-geo-ca_access    35         52149058
prod_in-ams03-geo-ca_access    99         52143277
prod_in-ams03-geo-ca_access    41         52158872
prod_in-ams03-geo-ca_access    112        52083901
prod_in-ams03-geo-ca_access    34         52137932
prod_in-ams03-geo-ca_access    89         52158135
prod_in-ams03-geo-ca_access    40         52144442
prod_in-ams03-geo-ca_access    53         52138400
prod_in-ams03-geo-ca_access    19         52144966
prod_in-ams03-geo-ca_access    44         52166404
prod_in-ams03-geo-ca_access    31         52155685
prod_in-ams03-geo-ca_access    10         52152151
prod_in-ams03-geo-ca_access    98         52145378
prod_in-ams03-geo-ca_access    69         52153436
prod_in-ams03-geo-ca_access    92         52093455
prod_in-ams03-geo-ca_access    5          52156448
prod_in-ams03-geo-ca_access    17         52063697
prod_in-ams03-geo-ca_access    24         52146410
prod_in-ams03-geo-ca_access    28         52156955
prod_in-ams03-geo-ca_access    82         52081194
prod_in-ams03-geo-ca_access    101        52153592
prod_in-ams03-geo-ca_access    58         52153765
prod_in-ams03-geo-ca_access    94         52162827
prod_in-ams03-geo-ca_access    116        52155307
prod_in-ams03-geo-ca_access    77         52072906
prod_in-ams03-geo-ca_access    68         52150283
prod_in-ams03-geo-ca_access    64         52155580
prod_in-ams03-geo-ca_access    109        52159915
prod_in-ams03-geo-ca_access    111        52153547
prod_in-ams03-geo-ca_access    2          52080434
prod_in-ams03-geo-ca_access    74         52148060
prod_in-ams03-geo-ca_access    47         52090187
prod_in-ams03-geo-ca_access    59         52145161
prod_in-ams03-geo-ca_access    1          52151228
prod_in-ams03-geo-ca_access    104        52159932
prod_in-ams03-geo-ca_access    14         52158366
prod_in-ams03-geo-ca_access    86         52153030
prod_in-ams03-geo-ca_access    81         52155507
prod_in-ams03-geo-ca_access    11         52158931
prod_in-ams03-geo-ca_access    100        52154646
prod_in-ams03-geo-ca_access    23         52169713
prod_in-ams03-geo-ca_access    45         52151320
prod_in-ams03-geo-ca_access    6          52157025
prod_in-ams03-geo-ca_access    87         52088913
prod_in-ams03-geo-ca_access    107        52086878
prod_in-ams03-geo-ca_access    16         52146481
prod_in-ams03-geo-ca_access    20         52148093
prod_in-ams03-geo-ca_access    103        52152437
prod_in-ams03-geo-ca_access    9          52158998
prod_in-ams03-geo-ca_access    29         52151112
prod_in-ams03-geo-ca_access    54         52149226
prod_in-ams03-geo-ca_access    36         52139520
prod_in-ams03-geo-ca_access    0          52152902
prod_in-ams03-geo-ca_access    84         52156057
prod_in-ams03-geo-ca_access    78         52155766
prod_in-ams03-geo-ca_access    118        52147043
prod_in-ams03-geo-ca_access    62         52087864
prod_in-ams03-geo-ca_access    108        52154615
prod_in-ams03-geo-ca_access    88         52147651
prod_in-ams03-geo-ca_access    39         52145255
prod_in-ams03-geo-ca_access    115        52166466
prod_in-ams03-geo-ca_access    51         52150136
prod_in-ams03-geo-ca_access    3          52147544
prod_in-ams03-geo-ca_access    73         52157150
prod_in-ams03-geo-ca_access    66         52158808
prod_in-ams03-geo-ca_access    67         52075616
prod_in-ams03-geo-ca_access    97         52086900
prod_in-ams03-geo-ca_access    15         52153225
prod_in-ams03-geo-ca_access    57         52097899
prod_in-ams03-geo-ca_access    8          52159097
prod_in-ams03-geo-ca_access    114        52149412
prod_in-ams03-geo-ca_access    37         52088584
prod_in-ams03-geo-ca_access    30         52153155
prod_in-ams03-geo-ca_access    79         52143773
prod_in-ams03-geo-ca_access    12         52089423
prod_in-ams03-geo-ca_access    21         52148187
prod_in-ams03-geo-ca_access    50         52150414
prod_in-ams03-geo-ca_access    60         52144624
prod_in-ams03-geo-ca_access    33         52149807
prod_in-ams03-geo-ca_access    46         52165008
prod_in-ams03-geo-ca_access    110        52153643
prod_in-ams03-geo-ca_access    95         52154736
prod_in-ams03-geo-ca_access    71         52150981
prod_in-ams03-geo-ca_access    96         52145091
prod_in-ams03-geo-ca_access    93         52149375
prod_in-ams03-geo-ca_access    91         52160207
prod_in-ams03-geo-ca_access    43         52151689
prod_in-ams03-geo-ca_access    63         52164909
prod_in-ams03-geo-ca_access    80         52151810
prod_in-ams03-geo-ca_access    70         52154522{code}
Then we ran the same command with the --execute option to the command to reset the offsets:
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access --execute{code}
And started the kafka mirrors for this topic.

However, it didn't work as expected for this topic in our production environment.
 The total log lag before resetting the offsets was 1,691,782,349:
{code:java}
root@ossmlbams0301a:/var/log/kafka# while true; do echo -n `date +'%Y-%m-%d %H:%M:%S'`; echo -n " "; /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --describe --group mirror-consumer-ams03-geo-earliest 2>/dev/null | awk '{if ($1 ~ "prod_in-ams03-geo-ca_access") { current_offset+=$3; end_offset+=$4; }} END{print "lag:", end_offset-current_offset;}'; sleep 15; done
2018-02-14 17:31:01 lag: 1691782349{code}
After resetting the offsets, the log lag dropped to 24,330:
{code:java}
2018-02-14 17:31:17 lag: 24330{code}
This was too much of a drop in the log lag.  This shows it did not really reset the offsets to start processing logs starting at Feb 10th (4 days ago).  We should be seeing a lag of around 37,170,000 if it worked correctly.  Instead, it seemed to reset the offsets to start processing logs starting today (Feb 14th).

We also tried resetting the offsets to start processing logs starting at February 8th (after stopping the mirrors) and it also did not work correctly.  Again, it seemed like it did a seek to the current date rather than resetting offsets to the date we specified.
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-08T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
Note: This will not show information about old Zookeeper-based consumers.

^@^@^@
TOPIC                          PARTITION  NEW-OFFSET
prod_in-ams03-geo-ca_access    52         52087106
prod_in-ams03-geo-ca_access    106        52156813
prod_in-ams03-geo-ca_access    75         52151755
prod_in-ams03-geo-ca_access    61         52133608
prod_in-ams03-geo-ca_access    49         52154385
prod_in-ams03-geo-ca_access    48         52148247
prod_in-ams03-geo-ca_access    27         52095874
prod_in-ams03-geo-ca_access    26         52142331
prod_in-ams03-geo-ca_access    65         52160181
prod_in-ams03-geo-ca_access    105        52169166
prod_in-ams03-geo-ca_access    38         52163198
prod_in-ams03-geo-ca_access    22         52096231
prod_in-ams03-geo-ca_access    4          52154512
prod_in-ams03-geo-ca_access    90         52163215
prod_in-ams03-geo-ca_access    25         52164527
prod_in-ams03-geo-ca_access    13         52148553
prod_in-ams03-geo-ca_access    56         52165558
prod_in-ams03-geo-ca_access    42         52075104
prod_in-ams03-geo-ca_access    7          52072333
prod_in-ams03-geo-ca_access    117        52090029
prod_in-ams03-geo-ca_access    32         52076469
prod_in-ams03-geo-ca_access    102        52085066
prod_in-ams03-geo-ca_access    76         52143820
prod_in-ams03-geo-ca_access    83         52157217
prod_in-ams03-geo-ca_access    72         52098060
prod_in-ams03-geo-ca_access    85         52152612
prod_in-ams03-geo-ca_access    119        52137225
prod_in-ams03-geo-ca_access    113        52162012
prod_in-ams03-geo-ca_access    55         52149399
prod_in-ams03-geo-ca_access    18         52151998
prod_in-ams03-geo-ca_access    35         52151756
prod_in-ams03-geo-ca_access    99         52146244
prod_in-ams03-geo-ca_access    41         52161567
prod_in-ams03-geo-ca_access    112        52086712
prod_in-ams03-geo-ca_access    34         52140841
prod_in-ams03-geo-ca_access    89         52160809
prod_in-ams03-geo-ca_access    40         52147177
prod_in-ams03-geo-ca_access    53         52141117
prod_in-ams03-geo-ca_access    19         52147752
prod_in-ams03-geo-ca_access    44         52169047
prod_in-ams03-geo-ca_access    31         52158455
prod_in-ams03-geo-ca_access    10         52155007
prod_in-ams03-geo-ca_access    98         52148061
prod_in-ams03-geo-ca_access    69         52156414
prod_in-ams03-geo-ca_access    92         52096106
prod_in-ams03-geo-ca_access    5          52159142
prod_in-ams03-geo-ca_access    17         52066397
prod_in-ams03-geo-ca_access    24         52149433
prod_in-ams03-geo-ca_access    28         52159909
prod_in-ams03-geo-ca_access    82         52083925
prod_in-ams03-geo-ca_access    101        52156350
prod_in-ams03-geo-ca_access    58         52156500
prod_in-ams03-geo-ca_access    94         52165527
prod_in-ams03-geo-ca_access    116        52158005
prod_in-ams03-geo-ca_access    77         52075690
prod_in-ams03-geo-ca_access    68         52152975
prod_in-ams03-geo-ca_access    64         52158277
prod_in-ams03-geo-ca_access    109        52162696
prod_in-ams03-geo-ca_access    111        52156535
prod_in-ams03-geo-ca_access    2          52083131
prod_in-ams03-geo-ca_access    74         52150817
prod_in-ams03-geo-ca_access    47         52092867
prod_in-ams03-geo-ca_access    59         52147851
prod_in-ams03-geo-ca_access    1          52154047
prod_in-ams03-geo-ca_access    104        52162658
prod_in-ams03-geo-ca_access    14         52161087
prod_in-ams03-geo-ca_access    86         52155828
prod_in-ams03-geo-ca_access    81         52158537
prod_in-ams03-geo-ca_access    11         52161597
prod_in-ams03-geo-ca_access    100        52157311
prod_in-ams03-geo-ca_access    23         52172470
prod_in-ams03-geo-ca_access    45         52154257
prod_in-ams03-geo-ca_access    6          52160103
prod_in-ams03-geo-ca_access    87         52091756
prod_in-ams03-geo-ca_access    107        52089608
prod_in-ams03-geo-ca_access    16         52149265
prod_in-ams03-geo-ca_access    20         52150732
prod_in-ams03-geo-ca_access    103        52155276
prod_in-ams03-geo-ca_access    9          52161987
prod_in-ams03-geo-ca_access    29         52153796
prod_in-ams03-geo-ca_access    54         52152245
prod_in-ams03-geo-ca_access    36         52142445
prod_in-ams03-geo-ca_access    0          52155910
prod_in-ams03-geo-ca_access    84         52159052
prod_in-ams03-geo-ca_access    78         52158632
prod_in-ams03-geo-ca_access    118        52149746
prod_in-ams03-geo-ca_access    62         52090560
prod_in-ams03-geo-ca_access    108        52157511
prod_in-ams03-geo-ca_access    88         52150632
prod_in-ams03-geo-ca_access    39         52148150
prod_in-ams03-geo-ca_access    115        52169203
prod_in-ams03-geo-ca_access    51         52153200
prod_in-ams03-geo-ca_access    3          52150514
prod_in-ams03-geo-ca_access    73         52159766
prod_in-ams03-geo-ca_access    66         52161783
prod_in-ams03-geo-ca_access    67         52078371
prod_in-ams03-geo-ca_access    97         52089805
prod_in-ams03-geo-ca_access    15         52156319
prod_in-ams03-geo-ca_access    57         52100913
prod_in-ams03-geo-ca_access    8          52161732
prod_in-ams03-geo-ca_access    114        52152390
prod_in-ams03-geo-ca_access    37         52091579
prod_in-ams03-geo-ca_access    30         52156230
prod_in-ams03-geo-ca_access    79         52146601
prod_in-ams03-geo-ca_access    12         52092444
prod_in-ams03-geo-ca_access    21         52151103
prod_in-ams03-geo-ca_access    50         52153126
prod_in-ams03-geo-ca_access    60         52147546
prod_in-ams03-geo-ca_access    33         52152862
prod_in-ams03-geo-ca_access    46         52167625
prod_in-ams03-geo-ca_access    110        52156369
prod_in-ams03-geo-ca_access    95         52157493
prod_in-ams03-geo-ca_access    71         52153659
prod_in-ams03-geo-ca_access    96         52148024
prod_in-ams03-geo-ca_access    93         52152429
prod_in-ams03-geo-ca_access    91         52162944
prod_in-ams03-geo-ca_access    43         52154651
prod_in-ams03-geo-ca_access    63         52167783
prod_in-ams03-geo-ca_access    80         52154510
prod_in-ams03-geo-ca_access    70         52157348{code}
If there is an invalid committed offset, could that maybe have caused it to not work correctly?  Even so, this should not be the result. 

We really want to be able to use this tool to reset offsets to a specified date.

Let me know if you need more information.

  was:
Using Kafka 1.0.0 we are seeing a problem using the kafka-consumer-groups.sh reset-offsets option to reset offsets to a specific date/time in our production environment.

We first tried to use the kafka-consumer-groups.sh command with the --reset-offsets option and with option --to-datetime 2018-02-10T00:00:00.000 in our staging environment and it worked correctly.  Running the following command changed it to start processing logs from February 12, 2018 (4 days ago) for a topic that had a large lag.  We did a dry run to verify before running with the --execute option.
{code:java}
root@ossmlpstagemon0101a:/# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.NNN.NN.NN:9092 --group logstash-elasticsearch-latest --to-datetime 2018-02-12T00:00:00.000-06:00 --reset-offsets --topic staging-mon01-rg-elasticsearch --execute{code}
We stopped the kafka mirrors that process this topic before resetting the offsets and started the kafka mirrors after rsetting the offsets.  We verified that it correctly started processing logs from February 12, 2018.

Then we tried resetting offsets in a production environment for a topic that had a very large lag using option --to-datetime 2018-02-10T00:00:00.000 and it did not work as expected. We stopped the kafka mirrors that process this topic before resetting the offsets and did a dry run to see what the new offsets would be:
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
Note: This will not show information about old Zookeeper-based consumers.

^@^@^@^@
TOPIC                          PARTITION  NEW-OFFSET
prod_in-ams03-geo-ca_access    52         52084147
prod_in-ams03-geo-ca_access    106        52154199
prod_in-ams03-geo-ca_access    75         52148673
prod_in-ams03-geo-ca_access    61         52130753
prod_in-ams03-geo-ca_access    49         52151667
prod_in-ams03-geo-ca_access    48         52145233
prod_in-ams03-geo-ca_access    27         52092805
prod_in-ams03-geo-ca_access    26         52139644
prod_in-ams03-geo-ca_access    65         52157504
prod_in-ams03-geo-ca_access    105        52166289
prod_in-ams03-geo-ca_access    38         52160464
prod_in-ams03-geo-ca_access    22         52093451
prod_in-ams03-geo-ca_access    4          52151660
prod_in-ams03-geo-ca_access    90         52160296
prod_in-ams03-geo-ca_access    25         52161691
prod_in-ams03-geo-ca_access    13         52145828
prod_in-ams03-geo-ca_access    56         52162867
prod_in-ams03-geo-ca_access    42         52072094
prod_in-ams03-geo-ca_access    7          52069496
prod_in-ams03-geo-ca_access    117        52087078
prod_in-ams03-geo-ca_access    32         52073732
prod_in-ams03-geo-ca_access    102        52082022
prod_in-ams03-geo-ca_access    76         52141018
prod_in-ams03-geo-ca_access    83         52154542
prod_in-ams03-geo-ca_access    72         52095051
prod_in-ams03-geo-ca_access    85         52149907
prod_in-ams03-geo-ca_access    119        52134435
prod_in-ams03-geo-ca_access    113        52159340
prod_in-ams03-geo-ca_access    55         52146597
prod_in-ams03-geo-ca_access    18         52149079
prod_in-ams03-geo-ca_access    35         52149058
prod_in-ams03-geo-ca_access    99         52143277
prod_in-ams03-geo-ca_access    41         52158872
prod_in-ams03-geo-ca_access    112        52083901
prod_in-ams03-geo-ca_access    34         52137932
prod_in-ams03-geo-ca_access    89         52158135
prod_in-ams03-geo-ca_access    40         52144442
prod_in-ams03-geo-ca_access    53         52138400
prod_in-ams03-geo-ca_access    19         52144966
prod_in-ams03-geo-ca_access    44         52166404
prod_in-ams03-geo-ca_access    31         52155685
prod_in-ams03-geo-ca_access    10         52152151
prod_in-ams03-geo-ca_access    98         52145378
prod_in-ams03-geo-ca_access    69         52153436
prod_in-ams03-geo-ca_access    92         52093455
prod_in-ams03-geo-ca_access    5          52156448
prod_in-ams03-geo-ca_access    17         52063697
prod_in-ams03-geo-ca_access    24         52146410
prod_in-ams03-geo-ca_access    28         52156955
prod_in-ams03-geo-ca_access    82         52081194
prod_in-ams03-geo-ca_access    101        52153592
prod_in-ams03-geo-ca_access    58         52153765
prod_in-ams03-geo-ca_access    94         52162827
prod_in-ams03-geo-ca_access    116        52155307
prod_in-ams03-geo-ca_access    77         52072906
prod_in-ams03-geo-ca_access    68         52150283
prod_in-ams03-geo-ca_access    64         52155580
prod_in-ams03-geo-ca_access    109        52159915
prod_in-ams03-geo-ca_access    111        52153547
prod_in-ams03-geo-ca_access    2          52080434
prod_in-ams03-geo-ca_access    74         52148060
prod_in-ams03-geo-ca_access    47         52090187
prod_in-ams03-geo-ca_access    59         52145161
prod_in-ams03-geo-ca_access    1          52151228
prod_in-ams03-geo-ca_access    104        52159932
prod_in-ams03-geo-ca_access    14         52158366
prod_in-ams03-geo-ca_access    86         52153030
prod_in-ams03-geo-ca_access    81         52155507
prod_in-ams03-geo-ca_access    11         52158931
prod_in-ams03-geo-ca_access    100        52154646
prod_in-ams03-geo-ca_access    23         52169713
prod_in-ams03-geo-ca_access    45         52151320
prod_in-ams03-geo-ca_access    6          52157025
prod_in-ams03-geo-ca_access    87         52088913
prod_in-ams03-geo-ca_access    107        52086878
prod_in-ams03-geo-ca_access    16         52146481
prod_in-ams03-geo-ca_access    20         52148093
prod_in-ams03-geo-ca_access    103        52152437
prod_in-ams03-geo-ca_access    9          52158998
prod_in-ams03-geo-ca_access    29         52151112
prod_in-ams03-geo-ca_access    54         52149226
prod_in-ams03-geo-ca_access    36         52139520
prod_in-ams03-geo-ca_access    0          52152902
prod_in-ams03-geo-ca_access    84         52156057
prod_in-ams03-geo-ca_access    78         52155766
prod_in-ams03-geo-ca_access    118        52147043
prod_in-ams03-geo-ca_access    62         52087864
prod_in-ams03-geo-ca_access    108        52154615
prod_in-ams03-geo-ca_access    88         52147651
prod_in-ams03-geo-ca_access    39         52145255
prod_in-ams03-geo-ca_access    115        52166466
prod_in-ams03-geo-ca_access    51         52150136
prod_in-ams03-geo-ca_access    3          52147544
prod_in-ams03-geo-ca_access    73         52157150
prod_in-ams03-geo-ca_access    66         52158808
prod_in-ams03-geo-ca_access    67         52075616
prod_in-ams03-geo-ca_access    97         52086900
prod_in-ams03-geo-ca_access    15         52153225
prod_in-ams03-geo-ca_access    57         52097899
prod_in-ams03-geo-ca_access    8          52159097
prod_in-ams03-geo-ca_access    114        52149412
prod_in-ams03-geo-ca_access    37         52088584
prod_in-ams03-geo-ca_access    30         52153155
prod_in-ams03-geo-ca_access    79         52143773
prod_in-ams03-geo-ca_access    12         52089423
prod_in-ams03-geo-ca_access    21         52148187
prod_in-ams03-geo-ca_access    50         52150414
prod_in-ams03-geo-ca_access    60         52144624
prod_in-ams03-geo-ca_access    33         52149807
prod_in-ams03-geo-ca_access    46         52165008
prod_in-ams03-geo-ca_access    110        52153643
prod_in-ams03-geo-ca_access    95         52154736
prod_in-ams03-geo-ca_access    71         52150981
prod_in-ams03-geo-ca_access    96         52145091
prod_in-ams03-geo-ca_access    93         52149375
prod_in-ams03-geo-ca_access    91         52160207
prod_in-ams03-geo-ca_access    43         52151689
prod_in-ams03-geo-ca_access    63         52164909
prod_in-ams03-geo-ca_access    80         52151810
prod_in-ams03-geo-ca_access    70         52154522{code}
Then we ran the same command with the --execute option to the command to reset the offsets:
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access --execute{code}
And started the kafka mirrors for this topic.

However, it didn't work as expected for this topic in our production environment.
 The total log lag before resetting the offsets was 1,691,782,349:
{code:java}
root@ossmlbams0301a:/var/log/kafka# while true; do echo -n `date +'%Y-%m-%d %H:%M:%S'`; echo -n " "; /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --describe --group mirror-consumer-ams03-geo-earliest 2>/dev/null | awk '{if ($1 ~ "prod_in-ams03-geo-ca_access") { current_offset+=$3; end_offset+=$4; }} END{print "lag:", end_offset-current_offset;}'; sleep 15; done
2018-02-14 17:31:01 lag: 1691782349{code}
After resetting the offsets, the log lag dropped drastically to 24,330:
{code:java}
2018-02-14 17:31:17 lag: 24330{code}
This was too much of a drop in the log lag which showed it did not really reset the offsets to start processing logs starting at Feb 10th (4 days ago).  We should be seeing a lag of around 37,170,000 if it worked correctly.  Instead, it seemed to reset the offsets to start processing logs starting today.

We also tried resetting the offsets to start processing logs starting at February 8th (after stopping the mirrors) and it also did not work correctly.  Again, it seemed like it did a seek to the current date rather than resetting offsets to the date we specified.
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-08T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
Note: This will not show information about old Zookeeper-based consumers.

^@^@^@
TOPIC                          PARTITION  NEW-OFFSET
prod_in-ams03-geo-ca_access    52         52087106
prod_in-ams03-geo-ca_access    106        52156813
prod_in-ams03-geo-ca_access    75         52151755
prod_in-ams03-geo-ca_access    61         52133608
prod_in-ams03-geo-ca_access    49         52154385
prod_in-ams03-geo-ca_access    48         52148247
prod_in-ams03-geo-ca_access    27         52095874
prod_in-ams03-geo-ca_access    26         52142331
prod_in-ams03-geo-ca_access    65         52160181
prod_in-ams03-geo-ca_access    105        52169166
prod_in-ams03-geo-ca_access    38         52163198
prod_in-ams03-geo-ca_access    22         52096231
prod_in-ams03-geo-ca_access    4          52154512
prod_in-ams03-geo-ca_access    90         52163215
prod_in-ams03-geo-ca_access    25         52164527
prod_in-ams03-geo-ca_access    13         52148553
prod_in-ams03-geo-ca_access    56         52165558
prod_in-ams03-geo-ca_access    42         52075104
prod_in-ams03-geo-ca_access    7          52072333
prod_in-ams03-geo-ca_access    117        52090029
prod_in-ams03-geo-ca_access    32         52076469
prod_in-ams03-geo-ca_access    102        52085066
prod_in-ams03-geo-ca_access    76         52143820
prod_in-ams03-geo-ca_access    83         52157217
prod_in-ams03-geo-ca_access    72         52098060
prod_in-ams03-geo-ca_access    85         52152612
prod_in-ams03-geo-ca_access    119        52137225
prod_in-ams03-geo-ca_access    113        52162012
prod_in-ams03-geo-ca_access    55         52149399
prod_in-ams03-geo-ca_access    18         52151998
prod_in-ams03-geo-ca_access    35         52151756
prod_in-ams03-geo-ca_access    99         52146244
prod_in-ams03-geo-ca_access    41         52161567
prod_in-ams03-geo-ca_access    112        52086712
prod_in-ams03-geo-ca_access    34         52140841
prod_in-ams03-geo-ca_access    89         52160809
prod_in-ams03-geo-ca_access    40         52147177
prod_in-ams03-geo-ca_access    53         52141117
prod_in-ams03-geo-ca_access    19         52147752
prod_in-ams03-geo-ca_access    44         52169047
prod_in-ams03-geo-ca_access    31         52158455
prod_in-ams03-geo-ca_access    10         52155007
prod_in-ams03-geo-ca_access    98         52148061
prod_in-ams03-geo-ca_access    69         52156414
prod_in-ams03-geo-ca_access    92         52096106
prod_in-ams03-geo-ca_access    5          52159142
prod_in-ams03-geo-ca_access    17         52066397
prod_in-ams03-geo-ca_access    24         52149433
prod_in-ams03-geo-ca_access    28         52159909
prod_in-ams03-geo-ca_access    82         52083925
prod_in-ams03-geo-ca_access    101        52156350
prod_in-ams03-geo-ca_access    58         52156500
prod_in-ams03-geo-ca_access    94         52165527
prod_in-ams03-geo-ca_access    116        52158005
prod_in-ams03-geo-ca_access    77         52075690
prod_in-ams03-geo-ca_access    68         52152975
prod_in-ams03-geo-ca_access    64         52158277
prod_in-ams03-geo-ca_access    109        52162696
prod_in-ams03-geo-ca_access    111        52156535
prod_in-ams03-geo-ca_access    2          52083131
prod_in-ams03-geo-ca_access    74         52150817
prod_in-ams03-geo-ca_access    47         52092867
prod_in-ams03-geo-ca_access    59         52147851
prod_in-ams03-geo-ca_access    1          52154047
prod_in-ams03-geo-ca_access    104        52162658
prod_in-ams03-geo-ca_access    14         52161087
prod_in-ams03-geo-ca_access    86         52155828
prod_in-ams03-geo-ca_access    81         52158537
prod_in-ams03-geo-ca_access    11         52161597
prod_in-ams03-geo-ca_access    100        52157311
prod_in-ams03-geo-ca_access    23         52172470
prod_in-ams03-geo-ca_access    45         52154257
prod_in-ams03-geo-ca_access    6          52160103
prod_in-ams03-geo-ca_access    87         52091756
prod_in-ams03-geo-ca_access    107        52089608
prod_in-ams03-geo-ca_access    16         52149265
prod_in-ams03-geo-ca_access    20         52150732
prod_in-ams03-geo-ca_access    103        52155276
prod_in-ams03-geo-ca_access    9          52161987
prod_in-ams03-geo-ca_access    29         52153796
prod_in-ams03-geo-ca_access    54         52152245
prod_in-ams03-geo-ca_access    36         52142445
prod_in-ams03-geo-ca_access    0          52155910
prod_in-ams03-geo-ca_access    84         52159052
prod_in-ams03-geo-ca_access    78         52158632
prod_in-ams03-geo-ca_access    118        52149746
prod_in-ams03-geo-ca_access    62         52090560
prod_in-ams03-geo-ca_access    108        52157511
prod_in-ams03-geo-ca_access    88         52150632
prod_in-ams03-geo-ca_access    39         52148150
prod_in-ams03-geo-ca_access    115        52169203
prod_in-ams03-geo-ca_access    51         52153200
prod_in-ams03-geo-ca_access    3          52150514
prod_in-ams03-geo-ca_access    73         52159766
prod_in-ams03-geo-ca_access    66         52161783
prod_in-ams03-geo-ca_access    67         52078371
prod_in-ams03-geo-ca_access    97         52089805
prod_in-ams03-geo-ca_access    15         52156319
prod_in-ams03-geo-ca_access    57         52100913
prod_in-ams03-geo-ca_access    8          52161732
prod_in-ams03-geo-ca_access    114        52152390
prod_in-ams03-geo-ca_access    37         52091579
prod_in-ams03-geo-ca_access    30         52156230
prod_in-ams03-geo-ca_access    79         52146601
prod_in-ams03-geo-ca_access    12         52092444
prod_in-ams03-geo-ca_access    21         52151103
prod_in-ams03-geo-ca_access    50         52153126
prod_in-ams03-geo-ca_access    60         52147546
prod_in-ams03-geo-ca_access    33         52152862
prod_in-ams03-geo-ca_access    46         52167625
prod_in-ams03-geo-ca_access    110        52156369
prod_in-ams03-geo-ca_access    95         52157493
prod_in-ams03-geo-ca_access    71         52153659
prod_in-ams03-geo-ca_access    96         52148024
prod_in-ams03-geo-ca_access    93         52152429
prod_in-ams03-geo-ca_access    91         52162944
prod_in-ams03-geo-ca_access    43         52154651
prod_in-ams03-geo-ca_access    63         52167783
prod_in-ams03-geo-ca_access    80         52154510
prod_in-ams03-geo-ca_access    70         52157348{code}
If there is an invalid committed offset, could that maybe have caused it to not work correctly?  Even so, this should not be the result. 

We really want to be able to use this tool to reset offsets to a specified date.


> kafka-consumer-groups does not reset offsets to specified datetime correctly
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-6572
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6572
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Sharon Lucas
>            Priority: Major
>
> With Kafka 1.0.0 we are seeing a problem using the kafka-consumer-groups.sh --reset-offsets option to reset offsets to a specific date/time in our production environment.
> We first tried to use the kafka-consumer-groups.sh command with the --reset-offsets option and with option --to-datetime 2018-02-10T00:00:00.000 in our staging environment and it worked correctly.  Running the following command changed it to start processing logs from February 12, 2018 (4 days ago) for a topic that had a large lag.  We did a dry run to verify before running with the --execute option.
> {code:java}
> root@ossmlpstagemon0101a:/# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.NNN.NN.NN:9092 --group logstash-elasticsearch-latest --to-datetime 2018-02-12T00:00:00.000-06:00 --reset-offsets --topic staging-mon01-rg-elasticsearch --execute{code}
> We stopped the kafka mirrors that process this topic before resetting the offsets and started the kafka mirrors after rsetting the offsets.  We verified that it correctly started processing logs from February 12, 2018.
> Then we tried resetting offsets in a production environment for a topic that had a very large lag using option --to-datetime 2018-02-10T00:00:00.000 and it did not work as expected. We stopped the kafka mirrors that process this topic before resetting the offsets and did a dry run to see what the new offsets would be:
> {code:java}
> root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
> Note: This will not show information about old Zookeeper-based consumers.
> ^@^@^@^@
> TOPIC                          PARTITION  NEW-OFFSET
> prod_in-ams03-geo-ca_access    52         52084147
> prod_in-ams03-geo-ca_access    106        52154199
> prod_in-ams03-geo-ca_access    75         52148673
> prod_in-ams03-geo-ca_access    61         52130753
> prod_in-ams03-geo-ca_access    49         52151667
> prod_in-ams03-geo-ca_access    48         52145233
> prod_in-ams03-geo-ca_access    27         52092805
> prod_in-ams03-geo-ca_access    26         52139644
> prod_in-ams03-geo-ca_access    65         52157504
> prod_in-ams03-geo-ca_access    105        52166289
> prod_in-ams03-geo-ca_access    38         52160464
> prod_in-ams03-geo-ca_access    22         52093451
> prod_in-ams03-geo-ca_access    4          52151660
> prod_in-ams03-geo-ca_access    90         52160296
> prod_in-ams03-geo-ca_access    25         52161691
> prod_in-ams03-geo-ca_access    13         52145828
> prod_in-ams03-geo-ca_access    56         52162867
> prod_in-ams03-geo-ca_access    42         52072094
> prod_in-ams03-geo-ca_access    7          52069496
> prod_in-ams03-geo-ca_access    117        52087078
> prod_in-ams03-geo-ca_access    32         52073732
> prod_in-ams03-geo-ca_access    102        52082022
> prod_in-ams03-geo-ca_access    76         52141018
> prod_in-ams03-geo-ca_access    83         52154542
> prod_in-ams03-geo-ca_access    72         52095051
> prod_in-ams03-geo-ca_access    85         52149907
> prod_in-ams03-geo-ca_access    119        52134435
> prod_in-ams03-geo-ca_access    113        52159340
> prod_in-ams03-geo-ca_access    55         52146597
> prod_in-ams03-geo-ca_access    18         52149079
> prod_in-ams03-geo-ca_access    35         52149058
> prod_in-ams03-geo-ca_access    99         52143277
> prod_in-ams03-geo-ca_access    41         52158872
> prod_in-ams03-geo-ca_access    112        52083901
> prod_in-ams03-geo-ca_access    34         52137932
> prod_in-ams03-geo-ca_access    89         52158135
> prod_in-ams03-geo-ca_access    40         52144442
> prod_in-ams03-geo-ca_access    53         52138400
> prod_in-ams03-geo-ca_access    19         52144966
> prod_in-ams03-geo-ca_access    44         52166404
> prod_in-ams03-geo-ca_access    31         52155685
> prod_in-ams03-geo-ca_access    10         52152151
> prod_in-ams03-geo-ca_access    98         52145378
> prod_in-ams03-geo-ca_access    69         52153436
> prod_in-ams03-geo-ca_access    92         52093455
> prod_in-ams03-geo-ca_access    5          52156448
> prod_in-ams03-geo-ca_access    17         52063697
> prod_in-ams03-geo-ca_access    24         52146410
> prod_in-ams03-geo-ca_access    28         52156955
> prod_in-ams03-geo-ca_access    82         52081194
> prod_in-ams03-geo-ca_access    101        52153592
> prod_in-ams03-geo-ca_access    58         52153765
> prod_in-ams03-geo-ca_access    94         52162827
> prod_in-ams03-geo-ca_access    116        52155307
> prod_in-ams03-geo-ca_access    77         52072906
> prod_in-ams03-geo-ca_access    68         52150283
> prod_in-ams03-geo-ca_access    64         52155580
> prod_in-ams03-geo-ca_access    109        52159915
> prod_in-ams03-geo-ca_access    111        52153547
> prod_in-ams03-geo-ca_access    2          52080434
> prod_in-ams03-geo-ca_access    74         52148060
> prod_in-ams03-geo-ca_access    47         52090187
> prod_in-ams03-geo-ca_access    59         52145161
> prod_in-ams03-geo-ca_access    1          52151228
> prod_in-ams03-geo-ca_access    104        52159932
> prod_in-ams03-geo-ca_access    14         52158366
> prod_in-ams03-geo-ca_access    86         52153030
> prod_in-ams03-geo-ca_access    81         52155507
> prod_in-ams03-geo-ca_access    11         52158931
> prod_in-ams03-geo-ca_access    100        52154646
> prod_in-ams03-geo-ca_access    23         52169713
> prod_in-ams03-geo-ca_access    45         52151320
> prod_in-ams03-geo-ca_access    6          52157025
> prod_in-ams03-geo-ca_access    87         52088913
> prod_in-ams03-geo-ca_access    107        52086878
> prod_in-ams03-geo-ca_access    16         52146481
> prod_in-ams03-geo-ca_access    20         52148093
> prod_in-ams03-geo-ca_access    103        52152437
> prod_in-ams03-geo-ca_access    9          52158998
> prod_in-ams03-geo-ca_access    29         52151112
> prod_in-ams03-geo-ca_access    54         52149226
> prod_in-ams03-geo-ca_access    36         52139520
> prod_in-ams03-geo-ca_access    0          52152902
> prod_in-ams03-geo-ca_access    84         52156057
> prod_in-ams03-geo-ca_access    78         52155766
> prod_in-ams03-geo-ca_access    118        52147043
> prod_in-ams03-geo-ca_access    62         52087864
> prod_in-ams03-geo-ca_access    108        52154615
> prod_in-ams03-geo-ca_access    88         52147651
> prod_in-ams03-geo-ca_access    39         52145255
> prod_in-ams03-geo-ca_access    115        52166466
> prod_in-ams03-geo-ca_access    51         52150136
> prod_in-ams03-geo-ca_access    3          52147544
> prod_in-ams03-geo-ca_access    73         52157150
> prod_in-ams03-geo-ca_access    66         52158808
> prod_in-ams03-geo-ca_access    67         52075616
> prod_in-ams03-geo-ca_access    97         52086900
> prod_in-ams03-geo-ca_access    15         52153225
> prod_in-ams03-geo-ca_access    57         52097899
> prod_in-ams03-geo-ca_access    8          52159097
> prod_in-ams03-geo-ca_access    114        52149412
> prod_in-ams03-geo-ca_access    37         52088584
> prod_in-ams03-geo-ca_access    30         52153155
> prod_in-ams03-geo-ca_access    79         52143773
> prod_in-ams03-geo-ca_access    12         52089423
> prod_in-ams03-geo-ca_access    21         52148187
> prod_in-ams03-geo-ca_access    50         52150414
> prod_in-ams03-geo-ca_access    60         52144624
> prod_in-ams03-geo-ca_access    33         52149807
> prod_in-ams03-geo-ca_access    46         52165008
> prod_in-ams03-geo-ca_access    110        52153643
> prod_in-ams03-geo-ca_access    95         52154736
> prod_in-ams03-geo-ca_access    71         52150981
> prod_in-ams03-geo-ca_access    96         52145091
> prod_in-ams03-geo-ca_access    93         52149375
> prod_in-ams03-geo-ca_access    91         52160207
> prod_in-ams03-geo-ca_access    43         52151689
> prod_in-ams03-geo-ca_access    63         52164909
> prod_in-ams03-geo-ca_access    80         52151810
> prod_in-ams03-geo-ca_access    70         52154522{code}
> Then we ran the same command with the --execute option to the command to reset the offsets:
> {code:java}
> root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access --execute{code}
> And started the kafka mirrors for this topic.
> However, it didn't work as expected for this topic in our production environment.
>  The total log lag before resetting the offsets was 1,691,782,349:
> {code:java}
> root@ossmlbams0301a:/var/log/kafka# while true; do echo -n `date +'%Y-%m-%d %H:%M:%S'`; echo -n " "; /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --describe --group mirror-consumer-ams03-geo-earliest 2>/dev/null | awk '{if ($1 ~ "prod_in-ams03-geo-ca_access") { current_offset+=$3; end_offset+=$4; }} END{print "lag:", end_offset-current_offset;}'; sleep 15; done
> 2018-02-14 17:31:01 lag: 1691782349{code}
> After resetting the offsets, the log lag dropped to 24,330:
> {code:java}
> 2018-02-14 17:31:17 lag: 24330{code}
> This was too much of a drop in the log lag.  This shows it did not really reset the offsets to start processing logs starting at Feb 10th (4 days ago).  We should be seeing a lag of around 37,170,000 if it worked correctly.  Instead, it seemed to reset the offsets to start processing logs starting today (Feb 14th).
> We also tried resetting the offsets to start processing logs starting at February 8th (after stopping the mirrors) and it also did not work correctly.  Again, it seemed like it did a seek to the current date rather than resetting offsets to the date we specified.
> {code:java}
> root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-08T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
> Note: This will not show information about old Zookeeper-based consumers.
> ^@^@^@
> TOPIC                          PARTITION  NEW-OFFSET
> prod_in-ams03-geo-ca_access    52         52087106
> prod_in-ams03-geo-ca_access    106        52156813
> prod_in-ams03-geo-ca_access    75         52151755
> prod_in-ams03-geo-ca_access    61         52133608
> prod_in-ams03-geo-ca_access    49         52154385
> prod_in-ams03-geo-ca_access    48         52148247
> prod_in-ams03-geo-ca_access    27         52095874
> prod_in-ams03-geo-ca_access    26         52142331
> prod_in-ams03-geo-ca_access    65         52160181
> prod_in-ams03-geo-ca_access    105        52169166
> prod_in-ams03-geo-ca_access    38         52163198
> prod_in-ams03-geo-ca_access    22         52096231
> prod_in-ams03-geo-ca_access    4          52154512
> prod_in-ams03-geo-ca_access    90         52163215
> prod_in-ams03-geo-ca_access    25         52164527
> prod_in-ams03-geo-ca_access    13         52148553
> prod_in-ams03-geo-ca_access    56         52165558
> prod_in-ams03-geo-ca_access    42         52075104
> prod_in-ams03-geo-ca_access    7          52072333
> prod_in-ams03-geo-ca_access    117        52090029
> prod_in-ams03-geo-ca_access    32         52076469
> prod_in-ams03-geo-ca_access    102        52085066
> prod_in-ams03-geo-ca_access    76         52143820
> prod_in-ams03-geo-ca_access    83         52157217
> prod_in-ams03-geo-ca_access    72         52098060
> prod_in-ams03-geo-ca_access    85         52152612
> prod_in-ams03-geo-ca_access    119        52137225
> prod_in-ams03-geo-ca_access    113        52162012
> prod_in-ams03-geo-ca_access    55         52149399
> prod_in-ams03-geo-ca_access    18         52151998
> prod_in-ams03-geo-ca_access    35         52151756
> prod_in-ams03-geo-ca_access    99         52146244
> prod_in-ams03-geo-ca_access    41         52161567
> prod_in-ams03-geo-ca_access    112        52086712
> prod_in-ams03-geo-ca_access    34         52140841
> prod_in-ams03-geo-ca_access    89         52160809
> prod_in-ams03-geo-ca_access    40         52147177
> prod_in-ams03-geo-ca_access    53         52141117
> prod_in-ams03-geo-ca_access    19         52147752
> prod_in-ams03-geo-ca_access    44         52169047
> prod_in-ams03-geo-ca_access    31         52158455
> prod_in-ams03-geo-ca_access    10         52155007
> prod_in-ams03-geo-ca_access    98         52148061
> prod_in-ams03-geo-ca_access    69         52156414
> prod_in-ams03-geo-ca_access    92         52096106
> prod_in-ams03-geo-ca_access    5          52159142
> prod_in-ams03-geo-ca_access    17         52066397
> prod_in-ams03-geo-ca_access    24         52149433
> prod_in-ams03-geo-ca_access    28         52159909
> prod_in-ams03-geo-ca_access    82         52083925
> prod_in-ams03-geo-ca_access    101        52156350
> prod_in-ams03-geo-ca_access    58         52156500
> prod_in-ams03-geo-ca_access    94         52165527
> prod_in-ams03-geo-ca_access    116        52158005
> prod_in-ams03-geo-ca_access    77         52075690
> prod_in-ams03-geo-ca_access    68         52152975
> prod_in-ams03-geo-ca_access    64         52158277
> prod_in-ams03-geo-ca_access    109        52162696
> prod_in-ams03-geo-ca_access    111        52156535
> prod_in-ams03-geo-ca_access    2          52083131
> prod_in-ams03-geo-ca_access    74         52150817
> prod_in-ams03-geo-ca_access    47         52092867
> prod_in-ams03-geo-ca_access    59         52147851
> prod_in-ams03-geo-ca_access    1          52154047
> prod_in-ams03-geo-ca_access    104        52162658
> prod_in-ams03-geo-ca_access    14         52161087
> prod_in-ams03-geo-ca_access    86         52155828
> prod_in-ams03-geo-ca_access    81         52158537
> prod_in-ams03-geo-ca_access    11         52161597
> prod_in-ams03-geo-ca_access    100        52157311
> prod_in-ams03-geo-ca_access    23         52172470
> prod_in-ams03-geo-ca_access    45         52154257
> prod_in-ams03-geo-ca_access    6          52160103
> prod_in-ams03-geo-ca_access    87         52091756
> prod_in-ams03-geo-ca_access    107        52089608
> prod_in-ams03-geo-ca_access    16         52149265
> prod_in-ams03-geo-ca_access    20         52150732
> prod_in-ams03-geo-ca_access    103        52155276
> prod_in-ams03-geo-ca_access    9          52161987
> prod_in-ams03-geo-ca_access    29         52153796
> prod_in-ams03-geo-ca_access    54         52152245
> prod_in-ams03-geo-ca_access    36         52142445
> prod_in-ams03-geo-ca_access    0          52155910
> prod_in-ams03-geo-ca_access    84         52159052
> prod_in-ams03-geo-ca_access    78         52158632
> prod_in-ams03-geo-ca_access    118        52149746
> prod_in-ams03-geo-ca_access    62         52090560
> prod_in-ams03-geo-ca_access    108        52157511
> prod_in-ams03-geo-ca_access    88         52150632
> prod_in-ams03-geo-ca_access    39         52148150
> prod_in-ams03-geo-ca_access    115        52169203
> prod_in-ams03-geo-ca_access    51         52153200
> prod_in-ams03-geo-ca_access    3          52150514
> prod_in-ams03-geo-ca_access    73         52159766
> prod_in-ams03-geo-ca_access    66         52161783
> prod_in-ams03-geo-ca_access    67         52078371
> prod_in-ams03-geo-ca_access    97         52089805
> prod_in-ams03-geo-ca_access    15         52156319
> prod_in-ams03-geo-ca_access    57         52100913
> prod_in-ams03-geo-ca_access    8          52161732
> prod_in-ams03-geo-ca_access    114        52152390
> prod_in-ams03-geo-ca_access    37         52091579
> prod_in-ams03-geo-ca_access    30         52156230
> prod_in-ams03-geo-ca_access    79         52146601
> prod_in-ams03-geo-ca_access    12         52092444
> prod_in-ams03-geo-ca_access    21         52151103
> prod_in-ams03-geo-ca_access    50         52153126
> prod_in-ams03-geo-ca_access    60         52147546
> prod_in-ams03-geo-ca_access    33         52152862
> prod_in-ams03-geo-ca_access    46         52167625
> prod_in-ams03-geo-ca_access    110        52156369
> prod_in-ams03-geo-ca_access    95         52157493
> prod_in-ams03-geo-ca_access    71         52153659
> prod_in-ams03-geo-ca_access    96         52148024
> prod_in-ams03-geo-ca_access    93         52152429
> prod_in-ams03-geo-ca_access    91         52162944
> prod_in-ams03-geo-ca_access    43         52154651
> prod_in-ams03-geo-ca_access    63         52167783
> prod_in-ams03-geo-ca_access    80         52154510
> prod_in-ams03-geo-ca_access    70         52157348{code}
> If there is an invalid committed offset, could that maybe have caused it to not work correctly?  Even so, this should not be the result. 
> We really want to be able to use this tool to reset offsets to a specified date.
> Let me know if you need more information.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Mime
View raw message