hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9782) RollingFileSystemSink should have configurable roll interval
Date Wed, 24 Feb 2016 23:05:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15165693#comment-15165693

Daniel Templeton commented on HDFS-9782:

Wow, checkstyle really doesn't like case statements to be indented...

Thanks for jumping in, [~andrew.wang]!

bq. If your concern is the linking between the interval and the offset, we could make the
offset configuration a percent of the interval.

My concern is that the offset interval alters when the metrics are reliably available.  I
think it violates the principal of least astonishment to have the metrics randomly (literally)
show up late by default.  I would rather it not be on unless it's needed, and the user turns
it on explicitly.

bq. I also agree with Robert and would prefer that we didn't add this unit parsing code at
all, but that's not a blocker.

What's the alternative?  I don't think millis is an acceptable unit for something that will
likely be hours or days.

bq. Also, if you look at BPServiceActor#Scheduler, this is an example of how we can unit test
a scheduler like this without sleeps. Food for thought.

Now I get what you meant in HDFS-9637 about testing using a clock that can be set by the tests.
 That seems pretty reasonable.  I clearly need to get better acquainted with Mockito.

I'll take another pass at it.

> RollingFileSystemSink should have configurable roll interval
> ------------------------------------------------------------
>                 Key: HDFS-9782
>                 URL: https://issues.apache.org/jira/browse/HDFS-9782
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Daniel Templeton
>            Assignee: Daniel Templeton
>         Attachments: HDFS-9782.001.patch, HDFS-9782.002.patch, HDFS-9782.003.patch, HDFS-9782.004.patch
> Right now it defaults to rolling at the top of every hour.  Instead that interval should
be configurable.  The interval should also allow for some play so that all hosts don't try
to flush their files simultaneously.
> I'm filing this in HDFS because I suspect it will involve touching the HDFS tests.  If
it turns out not to, I'll move it into common instead.

This message was sent by Atlassian JIRA

View raw message