hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-842) Serialize NN edits log as avro records
Date Tue, 12 Jan 2010 18:29:54 GMT

    [ https://issues.apache.org/jira/browse/HDFS-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799309#action_12799309
] 

Todd Lipcon commented on HDFS-842:
----------------------------------

Right, I don't think there's much doubt that this will be slower than the hand-crafted code
we've got now. However, the real question is what trade-off we're willing to make - as stated
above, this does bring us some good benefits in terms of data stability and portability. Let's
set out those goals before doing the work, so we can abandon early if it looks like we're
not going to hit them.

I'd propose that a 25% performance hit for loading and a 50% performance hit for saving are
reasonable targets. The reason I picked asymetric values is that loading is a critical bit
in the pipeline for today's NN recovery operation, whereas saving is done periodically and
so long as it's significantly faster than the inter-checkpoint time, it's not as important.

> Serialize NN edits log as avro records
> --------------------------------------
>
>                 Key: HDFS-842
>                 URL: https://issues.apache.org/jira/browse/HDFS-842
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: name-node
>            Reporter: Todd Lipcon
>
> Right now, the edits log is a mishmash of ad-hoc serialization and Writables. Switching
it over to Avro records would be really useful for operator tools - an "offline edits viewer"
would become trivial ("avrocat")

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message