accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1216) reconsider using SequenceFiles for the WAL
Date Thu, 28 Mar 2013 15:21:16 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13616341#comment-13616341
] 

Eric Newton commented on ACCUMULO-1216:
---------------------------------------

bq. It supports metadata, which we now have to do ourselves.

See o.a.a.server.tabletserver.DfsLogger.readHeader.  That code could go away if we used SequenceFile.
 This has not yet been released, so if we switched to SequenceFile this code would never have
to be supported in the future.

bq. There's some underlying machinery that can help with backward compatibility.

In the past, we have refactored and renamed the classes stored in the WAL.  Through the magic
of org.apache.hadoop.io.WritableName, it was pretty easy to read files that were written with
the old classnames.

In general, if I'm writing a sequence of Writables, and it has some magic at the beginning,
a bit of metadata, and it's going into HDFS, I think "SequenceFile."  I don't think "I'll
whip up my own file format."  Especially now that the reason we avoided SequenceFile might
be due to other misuses of HDFS  which have now been addressed.

Of course, we would have to reconsider how the encryption is wedged in there, but SequenceFile
supports a compression Codec, which might be an equivalent hook.

                
> reconsider using SequenceFiles for the WAL
> ------------------------------------------
>
>                 Key: ACCUMULO-1216
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1216
>             Project: Accumulo
>          Issue Type: Task
>          Components: tserver
>    Affects Versions: 1.5.0
>            Reporter: Eric Newton
>            Priority: Minor
>
> Observing the code in HBase we learned that WAL files written/flushed to HDFS would *not*
present the correct file size, which made using SequenceFile for WALs problematic.  So we
just write Writables.  In a sequence.
> However, it would be nice to go back and use SequenceFile.
> It might be possible since learning how to properly close a file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message