hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-842) Serialize NN edits log as avro records
Date Tue, 12 Jan 2010 18:37:54 GMT

    [ https://issues.apache.org/jira/browse/HDFS-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799311#action_12799311
] 

Eli Collins commented on HDFS-842:
----------------------------------

Since Avro can use compression I'm not sure it's going to be slower for a stream of edits.
And if we don't have many edits than it's probably not super relevant what the overhead is.
 Also currently syncing and loading logs (if there are multiple) does not happen in parallel,
doing that will provide a big speed up. ie we should pick up the low hanging fruit.  We should
still measure the impact of using avro here of course, just not sure that's were the significant
overhead is.

> Serialize NN edits log as avro records
> --------------------------------------
>
>                 Key: HDFS-842
>                 URL: https://issues.apache.org/jira/browse/HDFS-842
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: name-node
>            Reporter: Todd Lipcon
>
> Right now, the edits log is a mishmash of ad-hoc serialization and Writables. Switching
it over to Avro records would be really useful for operator tools - an "offline edits viewer"
would become trivial ("avrocat")

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message