spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raajay Viswanathan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-21334) Fix metrics for external shuffle service
Date Thu, 20 Jul 2017 17:38:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Raajay Viswanathan updated SPARK-21334:
---------------------------------------
    Description: 
SPARK-16405 introduced metrics for external shuffle service. However, as it is currently there
are two issues.

1. The shuffle service metrics system does not report values ever.
-2. The current method for determining "blockTransferRate" is incorrect. The entire block
is assumed to be transferred once the OpenBlocks message if processed. The actual data fetch
from the disk and the succeeding transfer over the wire happens much later when MessageEncoder
invokes encode on ChunkFetchSuccess message. -


  was:
SPARK-16405 introduced metrics for external shuffle service. However, as it is currently there
are two issues.

1. The shuffle service metrics system does not report values ever.
2. The current method for determining "blockTransferRate" is incorrect. The entire block is
assumed to be transferred once the OpenBlocks message if processed. The actual data fetch
from the disk and the succeeding transfer over the wire happens much later when MessageEncoder
invokes encode on ChunkFetchSuccess message. 



> Fix metrics for external shuffle service
> ----------------------------------------
>
>                 Key: SPARK-21334
>                 URL: https://issues.apache.org/jira/browse/SPARK-21334
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>    Affects Versions: 2.1.1
>            Reporter: Raajay Viswanathan
>              Labels: external-shuffle-service
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> SPARK-16405 introduced metrics for external shuffle service. However, as it is currently
there are two issues.
> 1. The shuffle service metrics system does not report values ever.
> -2. The current method for determining "blockTransferRate" is incorrect. The entire block
is assumed to be transferred once the OpenBlocks message if processed. The actual data fetch
from the disk and the succeeding transfer over the wire happens much later when MessageEncoder
invokes encode on ChunkFetchSuccess message. -



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message