ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: IGFS failover: add a field into IgfsFileInfo: question on backwards compatibility.
Date Fri, 28 Aug 2015 18:27:07 GMT
May be I am missing something but here's how HDFS works at this level (in a
very drafty description)...

 - a client ask NN to create new file
 - a client asks for a first block and starts writing the data into a
   designated DN
 - the data is written in chunks, that are reported to the NN to record
   'available to read' length. It is more complex that just that because
   involves a DN-pipelining, but isn't material for this conversation
 - if a client crashes in a middle of the writing, the lease will remain and
   can be recovered if a client were to re-connect and continued the writing
 - if client never comes back the block replicas will remain incomplete and NN
   will treat them accordingly

Basically, meta and data are always written separately by design.  Looks like
you want to implement an improvement on top of what is already there, if I
read this correctly. Perhaps, the JIRA needs a bit more detailed description
or this is just me?


On Fri, Aug 28, 2015 at 05:21PM, Ivan Veselovskiy wrote:
> Hi, colleagues,
> working on IGFS failover functionality (IGNITE-1298, IGNITE-1277) we
> encountered the following issue: we would like to track IGFS meta cache
> updates in Ignite cluster to be able to track write progress of IGFS files
> -- this will allow us to clean up garbage data blocks belonging to
> partially written files when a writing node has crashed.
> To do that we would like to add a new field into IgfsFileInfo class, but
> that will break backwards compatibility in case old node works with new
> nodes in the same cluster, because of [de]serialization issues.
> So, the question is: is that okay to break the compatibility in such a way?
> If not, what is the nearest major release where this compatibility can be
> broken?
> Thanks.
> --ivan

View raw message