hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: A question for txid
Date Tue, 25 Jun 2013 13:14:32 GMT
Yes, it logically can if there have been as many transactions (its a
very very large number to reach though).

Long.MAX_VALUE is (2^63 - 1) or 9223372036854775807.

I hacked up my local NN's txids manually to go very large (close to
max) and decided to try out if this causes any harm. I basically
bumped up the freshly formatted starting txid to 9223372036854775805
(and ensured image references the same):

➜  current  ls
➜  current  cat seen_txid

NameNode started up as expected.

13/06/25 18:30:08 INFO namenode.FSImage: Image file of size 129 loaded
in 0 seconds.
13/06/25 18:30:08 INFO namenode.FSImage: Loaded image for txid
9223372036854775805 from
13/06/25 18:30:08 INFO namenode.FSEditLog: Starting log segment at

I could create a bunch of files and do regular ops (counting to much
after the long max increments). I created over 100 files, just to make
it go well over the Long.MAX_VALUE.

Quitting NameNode and restarting fails though, with the following error:

13/06/25 18:31:08 FATAL namenode.NameNode: Exception in namenode join
java.io.IOException: Gap in transactions. Expected to be able to read
up until at least txid 9223372036854775806 but unable to find any edit
logs containing txid -9223372036854775808

So it looks like it cannot currently handle an overflow.

I've filed https://issues.apache.org/jira/browse/HDFS-4936 to discuss
this. I don't think this is of immediate concern though, so we should
be able to address it in future (unless there's parts of the code
which already are preventing reaching this number in the first place -
please do correct me if there is such a part).

On Tue, Jun 25, 2013 at 3:09 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:
> Hi dear All,
> It's long type for the txid currently,
> FSImage.java:
> boolean loadFSImage(FSNamesystem target, MetaRecoveryContext recovery)
>     throws IOException{
>   editLog.setNextTxId(lastAppliedTxId + 1L);
> }
> Is it possible that (lastAppliedTxId + 1L) exceed Long.MAX_VALUE ?

Harsh J

View raw message