hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13028) add low level counter metrics for S3A; use in read performance tests
Date Fri, 06 May 2016 18:50:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274562#comment-15274562
] 

Chris Nauroth commented on HADOOP-13028:
----------------------------------------

Patch v010 addresses all of my prior feedback.

It's fine to keep {{S3AInstrumentation#gauge}}.

I agree that "must be private and have accessor methods" is pedantic.  I think we can control
this with the Checkstyle [HiddenField|http://checkstyle.sourceforge.net/config_coding.html#HiddenField]
rule settings.  Let's not bother for this patch though.  You've already done a lot to clean
up Checkstyle warnings.

I agree with updating the JIRA title and closing down the other ones that were folded into
this patch.

I'm now seeing a failure in {{TestS3ABlocksize#testBlockSize}}:
{code}
testBlockSize(org.apache.hadoop.fs.s3a.TestS3ABlocksize)  Time elapsed: 3.17 sec  <<<
FAILURE!
java.lang.AssertionError: Double default block size in stat(): S3AFileStatus{path=s3a://cnauroth-test-aws-s3a/test/testBlockSize/file;
isDirectory=false; length=1024; replication=1; blocksize=33554432; modification_time=1462559933000;
access_time=0; owner=; group=; permission=rw-rw-rw-; isSymlink=false} expected:<67108864>
but was:<33554432>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.apache.hadoop.fs.s3a.TestS3ABlocksize.testBlockSize(TestS3ABlocksize.java:65)
{code}

It looks like the test has an expectation that you can update block size in the {{Configuration}}
of an already-initialized {{S3AFileSystem}}, and it will start using the new value.  After
this patch, the block size is read from {{Configuration}} once and cached.  I think the expectation
of the test is a little dubious, but that's the behavior that shipped in 2.7.0.  It's probably
safest to remove this part of the patch for now.

I just have one more request.  Please remove the unused import of {{Configuration}} in {{S3AInstrumentation}}.

I'll be +1 after that.  We'll also need a separate trunk patch.

> add low level counter metrics for S3A; use in read performance tests
> --------------------------------------------------------------------
>
>                 Key: HADOOP-13028
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13028
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, metrics
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, HADOOP-13028-004.patch,
HADOOP-13028-005.patch, HADOOP-13028-006.patch, HADOOP-13028-007.patch, HADOOP-13028-008.patch,
HADOOP-13028-009.patch, HADOOP-13028-branch-2-008.patch, HADOOP-13028-branch-2-009.patch,
HADOOP-13028-branch-2-010.patch, org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt,
org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive, closing connections
may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of open/close/failure+reconnect
operations, timers of how long things take. This can be used downstream to measure efficiency
of the code (how often connections are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message