hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-842) Serialize NN edits log as avro records
Date Tue, 12 Jan 2010 18:41:54 GMT

    [ https://issues.apache.org/jira/browse/HDFS-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799313#action_12799313
] 

Todd Lipcon commented on HDFS-842:
----------------------------------

In my experience I've usually seen NN edit log replay to be CPU bound, not IO bound, so I
don't anticipate the compression will make a huge difference. We can't make the replay itself
multithreaded since we need the right order, but we could certainly multithread the deserialization,
in which case we're just soaking up previously unused CPU resources - that's what you're getting
at?

> Serialize NN edits log as avro records
> --------------------------------------
>
>                 Key: HDFS-842
>                 URL: https://issues.apache.org/jira/browse/HDFS-842
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: name-node
>            Reporter: Todd Lipcon
>
> Right now, the edits log is a mishmash of ad-hoc serialization and Writables. Switching
it over to Avro records would be really useful for operator tools - an "offline edits viewer"
would become trivial ("avrocat")

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message