hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14425) Add more s3guard metrics
Date Thu, 18 May 2017 10:55:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015573#comment-16015573
] 

Steve Loughran commented on HADOOP-14425:
-----------------------------------------

BTW, dump of stats on a test run without s3guard enabled; committer stats added
{code}
2017-05-18 11:53:05,631 [JUnit-ITestStagingCommitProtocol-testCommitLifecycle] INFO  commit.AbstractITCommitProtocol
(AbstractITCommitProtocol.java:teardown(169)) - Statistics for s3a://hwdev-steve-ireland-new:
  Context =  S3AFileSystem
  FileSystemId =  fc085a79-a825-4edf-9c99-bd1d5afead56-hwdev-steve-ireland-new
  fsURI =  s3a://hwdev-steve-ireland-new/
  files_created =  1
  files_copied =  0
  files_copied_bytes =  0
  files_deleted =  2
  fake_directories_deleted =  4
  directories_created =  2
  directories_deleted =  0
  ignored_errors =  3
  op_copy_from_local_file =  0
  op_exists =  0
  op_get_file_status =  11
  op_glob_status =  0
  op_is_directory =  0
  op_is_file =  0
  op_list_files =  1
  op_list_located_status =  0
  op_list_status =  1
  op_mkdirs =  1
  op_rename =  0
  object_copy_requests =  0
  object_delete_requests =  3
  object_list_requests =  12
  object_continue_list_requests =  0
  object_metadata_requests =  20
  object_multipart_aborted =  0
  object_put_bytes =  2897
  object_put_requests =  4
  object_put_requests_completed =  4
  stream_write_failures =  0
  stream_write_block_uploads =  0
  stream_write_block_uploads_committed =  0
  stream_write_block_uploads_aborted =  0
  stream_write_total_time =  0
  stream_write_total_data =  2857
  s3guard_committer_commits_created =  0
  s3guard_committer_commits_completed =  1
  s3guard_committer_jobs_completed =  1
  s3guard_committer_jobs_failed =  0
  s3guard_committer_tasks_completed =  1
  s3guard_committer_tasks_failed =  0
  s3guard_committer_bytes_committed =  1
  s3guard_committer_commits_failed =  0
  s3guard_committer_commits_aborted =  0
  s3guard_committer_commits_reverted =  0
  s3guard_metadatastore_put_path_request =  5
  s3guard_metadatastore_initialization =  0
  object_put_requests_active =  0
  object_put_bytes_pending =  0
  stream_write_block_uploads_active =  0
  stream_write_block_uploads_pending =  1
  stream_write_block_uploads_data_pending =  0
  S3guard_metadatastore_put_path_latencyNumOps =  4
  S3guard_metadatastore_put_path_latency50thPercentileLatency =  5384
  S3guard_metadatastore_put_path_latency75thPercentileLatency =  5950
  S3guard_metadatastore_put_path_latency90thPercentileLatency =  5950
  S3guard_metadatastore_put_path_latency95thPercentileLatency =  5950
  S3guard_metadatastore_put_path_latency99thPercentileLatency =  5950
  stream_read_fully_operations =  1
  stream_opened =  1
  stream_bytes_skipped_on_seek =  0
  stream_closed =  1
  stream_bytes_backwards_on_seek =  0
  stream_bytes_read =  40
  stream_read_operations_incomplete =  0
  stream_bytes_discarded_in_abort =  0
  stream_close_operations =  1
  stream_read_operations =  1
  stream_aborted =  0
  stream_forward_seek_operations =  0
  stream_backward_seek_operations =  0
  stream_seek_operations =  0
  stream_bytes_read_in_close =  0
  stream_read_exceptions =  0
{code}

> Add more s3guard metrics
> ------------------------
>
>                 Key: HADOOP-14425
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14425
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Ai Deng
>
> The metrics suggested to add:
> Status:
> S3GUARD_METADATASTORE_ENABLED
> S3GUARD_METADATASTORE_IS_AUTHORITATIVE
> Operations:
> S3GUARD_METADATASTORE_INITIALIZATION
> S3GUARD_METADATASTORE_DELETE_PATH
> S3GUARD_METADATASTORE_DELETE_PATH_LATENCY
> S3GUARD_METADATASTORE_DELETE_SUBTREE_PATCH
> S3GUARD_METADATASTORE_GET_PATH
> S3GUARD_METADATASTORE_GET_PATH_LATENCY
> S3GUARD_METADATASTORE_GET_CHILDREN_PATH
> S3GUARD_METADATASTORE_GET_CHILDREN_PATH_LATENCY
> S3GUARD_METADATASTORE_MOVE_PATH
> S3GUARD_METADATASTORE_PUT_PATH
> S3GUARD_METADATASTORE_PUT_PATH_LATENCY
> S3GUARD_METADATASTORE_CLOSE
> S3GUARD_METADATASTORE_DESTORY
> From S3Guard:
> S3GUARD_METADATASTORE_MERGE_DIRECTORY
> For the failures:
> S3GUARD_METADATASTORE_DELETE_FAILURE
> S3GUARD_METADATASTORE_GET_FAILURE
> S3GUARD_METADATASTORE_PUT_FAILURE
> Etc:
> S3GUARD_METADATASTORE_PUT_RETRY_TIMES



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message