bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Olivelli - Diennea <enrico.olive...@diennea.com>
Subject R: Journal file never deleted
Date Tue, 08 Sep 2015 12:51:33 GMT
I think I go into
https://issues.apache.org/jira/browse/BOOKKEEPER-695



Da: Enrico Olivelli - Diennea [mailto:enrico.olivelli@diennea.com]
Inviato: lunedì 13 luglio 2015 12:58
A: user@bookkeeper.apache.org
Oggetto: R: Journal file never deleted

At time of writing in my test env I have:

-          132 .txn files

-          Most of them are 16777216 bytes

-          I have 0 ledgers

-          journalMaxSizeMB=1048

-          logSizeLimit (entryLogSizeLimit)= 2147483648

-          size of directory referred by ledgerDirNames is 2.8MB

honestly I can’t estimate the effective size of data which was written to my bookie
I do not have any automatic test case useful to reproduce the problem

The procedure is simple, the are two main use cases:

1)      Open ledger and write…..close. drop it  , close le client  (most common case of
course)

2)      Open ledger and write, if an error occurs (on other resource or during writes to BK),
try to close the ledger, try to close the client, then from another procedure (usually on
another machine) open the ledger (with fencing), read all the entries, close the ledger, drop
the ledger

a.       Important Subcase: errors during the “recovery”…then we need to retry recovery
after some time



Da: Sijie Guo [mailto:sijie@apache.org]
Inviato: giovedì 9 luglio 2015 18:30
A: user@bookkeeper.apache.org<mailto:user@bookkeeper.apache.org>; Enrico Olivelli -
Diennea
Oggetto: Re: Journal file never deleted

The org.apache.bookkeeper.bookie.CheckpointSource$Checkpoint are the inner classes 'MAX' and
'MIN'. They are expected if entry log file isn't rotated. As the checkpoint logic in bookie
is only checkpoint when entry log file rotated, for performance consideration.
I am not sure if we are on same page. deleting ledgers & stopping writing entries won't
help deleting journal files. The way how journal files rotated and deleted works as below:
- all entries are appended in journal
- entries are added into ledger storage and periodic flushed to disk.
- once the ledger storage is flushed, the log mark will be updated, indicating entries added
before the mark are persisted. then the journals before the mark is able to be deleted.
- so if there is only one journal, it won't be deleted. if there are more journal files, it
would only delete journals that before log mark and keep  #getMaxBackupJournals journals before
log mark.
so it would be good that you could share more details on the problem you encountered:
- how many journal files you have?
- how many entry log files (.log) you have?
- what is the your journal max size? 'journalMaxSizeMB'
- what is your size of entry log file? 'logSizeLimit'
- how many data you have written?
- how's your procedure on reproducing this behavior?
- Sijie


On Thu, Jul 9, 2015 at 12:13 AM, Enrico Olivelli - Diennea <enrico.olivelli@diennea.com<mailto:enrico.olivelli@diennea.com>>
wrote:
I reproduced the same problem in a simple staging environment, I deleted every ledger and
stopped writing new entries.

I’m trying to do a little debug and I see that in Journal.java  there is this line
if (!(checkpoint instanceof LogMarkCheckpoint)) {
            return; // we didn't create this checkpoint, so dont do anything with it
        }

Actually my test bookie gets a org.apache.bookkeeper.bookie.CheckpointSource$Checkpoint$1@98aabc2<mailto:org.apache.bookkeeper.bookie.CheckpointSource$Checkpoint$1@98aabc2>
class instead of a LogMarkCheclpoint, when the
checkpoint is done in the SynchThread#checkpoint method at every FlushInterval

hope this can help

Enrico


Da: Sijie Guo [mailto:sijie@apache.org<mailto:sijie@apache.org>]
Inviato: mercoledì 8 luglio 2015 19:06

A: user@bookkeeper.apache.org<mailto:user@bookkeeper.apache.org>
Oggetto: Re: Journal file never deleted

The journal files are only deleted when its corresponding data in ledger storage is flushed.
What is your rate on adding entries? How many .txn files you have?

On Wed, Jul 8, 2015 at 3:27 AM, Enrico Olivelli - Diennea <enrico.olivelli@diennea.com<mailto:enrico.olivelli@diennea.com>>
wrote:
I’m running all the default values, except from maxbackups which I set to 1
Even if a delete every ledger the logs never get deleted.

I enabled level DEBUG for every logger but I get no useful hint.
Normally at which rate logs file should be deleted ?

Thanks
Enrico


Da: Flavio Junqueira [mailto:fpjunqueira@yahoo.com<mailto:fpjunqueira@yahoo.com>]
Inviato: mercoledì 8 luglio 2015 12:17
A: user@bookkeeper.apache.org<mailto:user@bookkeeper.apache.org>
Oggetto: Re: Journal file never deleted

Hi Enrico,

Have you had a look at the journal settings here?

http://bookkeeper.apache.org/docs/r4.3.1/bookieConfigParams.html

-Flavio

On 08 Jul 2015, at 10:37, Enrico Olivelli - Diennea <enrico.olivelli@diennea.com<mailto:enrico.olivelli@diennea.com>>
wrote:

Hi,
I’m running BookKeeper 4.3.0 (and 4.3.1). I noticed that journal files (*.txn files) never
get deleted. I’m using ServerConfiguration.setMaxBackupJournals(1) and default flushinterval
of 1 second.
I’m running “embedded” bookies, that is that I launch the bookie inside another JVM,
with the following code:

ServerConfiguration   conf = new ServerConfiguration();
conf.setBookiePort(port);
conf.setUseHostNameAsBookieID(true);
conf.setJournalDirName(logDir.getAbsolutePath());
conf.setZkServers(zookeeperServer);
conf.setLedgerDirNames(Arrays.stream(dataDirs).map(File::getAbsolutePath).collect(Collectors.toList()).toArray(new
String[dataDirs.length]));
conf.setFlushInterval(1000);
conf.setAutoRecoveryDaemonEnabled(false);
…
conf.setMaxBackupJournals(1);
…
this.bookie = new BookieServer(conf);
this.bookee.start();

any suggestion ?

thanks

Enrico Olivelli
Software Development Manager @Diennea
Tel.: (+39) 0546 066100<tel:%28%2B39%29%C2%A00546%C2%A0066100> - Int. 925
Viale G.Marconi 30/14 - 48018 Faenza (RA)

MagNews - E-mail Marketing Solutions
http://www.magnews.it<http://www.magnews.it/>
Diennea - Digital Marketing Solutions
http://www.diennea.com<http://www.diennea.com/>


________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/


________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/


________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/


________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/

________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/
Mime
View raw message