kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yang Ye (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-593) Empty log index file created when it shouldn't be empty
Date Fri, 02 Nov 2012 23:42:12 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yang Ye updated KAFKA-593:
--------------------------

    Attachment: kafka_593_v2.diff


Jay, Jun,

Thanks for the comments.

Jun also pointed out that there may still be cases where at startup, the last log segment
has empty index and empty log file ---- and the trimOrReallocate() is not called because it
was a clean shutdown before. 

What about an alternative idea, which is, whenever we load a log segment, we always make sure
its offset index has enough disk space and memory. In this case, when we truncate back to
old segment, its index will not be full, when we start the last log segment with empty log
file and index file, its index will also not be full. 

To do this, we just need to change the constructor of OffsetIndex, by always set the index
file size and mmap limit to maxIndexSize.

                
> Empty log index file created when it shouldn't be empty
> -------------------------------------------------------
>
>                 Key: KAFKA-593
>                 URL: https://issues.apache.org/jira/browse/KAFKA-593
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Yang Ye
>         Attachments: kafka_583_zk_kafka_data.tar.gz, kafka_593_v1.diff, kafka_593_v2.diff
>
>
> We have met empty index file during system test when it shouldn't be empty. In this case,
there're around 100 messages in each segment, each of size around 100 bytes, given the "logIndexIntervalBytes"
4096, there should be at least 2 log index entries, but we see empty index file. The kafka
and zookeeper logs are attached
> [yye@yye-ld kafka_server_3_logs]$ cd test_1-2/
> [yye@yye-ld test_1-2]$ ls -l
> total 84
> -rw-r--r-- 1 yye eng        8 Oct 29 15:22 00000000000000000000.index
> -rw-r--r-- 1 yye eng    10248 Oct 29 15:22 00000000000000000000.log
> -rw-r--r-- 1 yye eng        8 Oct 29 15:22 00000000000000000100.index
> -rw-r--r-- 1 yye eng    10296 Oct 29 15:22 00000000000000000100.log
> -rw-r--r-- 1 yye eng        0 Oct 29 15:23 00000000000000000200.index
> -rw-r--r-- 1 yye eng    10293 Oct 29 15:23 00000000000000000200.log
> -rw-r--r-- 1 yye eng        0 Oct 29 15:23 00000000000000000300.index
> -rw-r--r-- 1 yye eng    10274 Oct 29 15:23 00000000000000000300.log
> -rw-r--r-- 1 yye eng        0 Oct 29 15:23 00000000000000000399.index
> -rw-r--r-- 1 yye eng    10276 Oct 29 15:23 00000000000000000399.log
> -rw-r--r-- 1 yye eng        0 Oct 29 15:23 00000000000000000498.index
> -rw-r--r-- 1 yye eng    10256 Oct 29 15:23 00000000000000000498.log
> -rw-r--r-- 1 yye eng 10485760 Oct 29 15:23 00000000000000000596.index
> -rw-r--r-- 1 yye eng     3564 Oct 29 15:23 00000000000000000596.log

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message