spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thunder Stumpges (JIRA)" <j...@apache.org>
Subject [jira] (SPARK-19371) Cannot spread cached partitions evenly across executors
Date Mon, 30 Jan 2017 16:32:43 GMT
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml"> 
    <head> 
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 
        <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0"
/> <base href="https://issues.apache.org/jira" /> 
        <title>Message Title</title> 
    </head> 
    <body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: 14px;
line-height: 1.429"> 
        <table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
            <!-- header here --> 
            <tr> 
                <td id="header-pattern-container" style="padding: 0px; border-collapse:
collapse; padding: 10px 20px"> 
                    <table id="header-pattern" cellspacing="0" cellpadding="0" border="0"
style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
                        <tr> 
                            <td id="header-avatar-image-container" valign="top" style="padding:
0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px">
<img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-thunderstumpges-310a8474-1294-4ca5-b4dd-e035cb8d885a"
height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" /> 
                            </td> 
                            <td id="header-text-container" valign="middle" style="padding:
0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover"
rel="thunderstumpges" id="email_thunderstumpges" href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=thunderstumpges"
style="color:#3b73af;; color: #3b73af; text-decoration: none">Thunder Stumpges</a>
<strong>edited a comment</strong> on <a href="https://issues.apache.org/jira/browse/SPARK-19371"
style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-static-bug-3eda9689-d125-4aca-bb89-9285d00b9a14"
height="16" width="16" border="0" align="absmiddle" alt="Bug" /> SPARK-19371</a>

                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
            <tr> 
                <td id="email-content-container" style="padding: 0px; border-collapse:
collapse; padding: 0 20px"> 
                    <table id="email-content-table" cellspacing="0" cellpadding="0" border="0"
width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt;
border-spacing: 0; border-collapse: separate"> 
                        <tr> 
                            <!-- there needs to be content in the cell for it to render
in some clients --> 
                            <td class="email-content-rounded-top mobile-expand" style="padding:
0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 15px; background-color:
#fff; border-left: 1px solid #ccc; border-top: 1px solid #ccc; border-right: 1px solid #ccc;
border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px;
line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly">
                                &nbsp;
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand " style="padding:
0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc;
border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff"> 
                                <table class="page-title-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt"> 
                                    <tr> 
                                        <td style="vertical-align: top;; padding: 0px;
border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule:
exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header"
style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise:
2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.apache.org/jira/browse/SPARK-19371"
style="color: #3b73af; text-decoration: none">Re: Cannot spread cached partitions evenly
across executors</a> </span> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <tr> 
                            <td id="text-paragraph-pattern-top" class="email-content-main
mobile-expand  comment-top-special-margin comment-top-pattern" style="padding: 0px; border-collapse:
collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom:
0; padding: 0 15px 0 16px; background-color: #fff; border-bottom: none; padding-bottom: 0">

                                <table class="text-paragraph-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule:
exactly; mso-text-raise: 2px"> 
                                    <tr> 
                                        <td class="text-paragraph-pattern-container mobile-resize-text
" style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0; padding-top: 10px">
<span class="diffcontext">{quote}<br />...&nbsp;one&nbsp;execution&nbsp;slot&nbsp;is&nbsp;as&nbsp;good&nbsp;as&nbsp;another,&nbsp;because&nbsp;it's&nbsp;a&nbsp;core&nbsp;operating&nbsp;on&nbsp;local&nbsp;memory.&nbsp;I&nbsp;do&nbsp;believe&nbsp;Spark&nbsp;already&nbsp;prefers&nbsp;spreading&nbsp;tasks&nbsp;across&nbsp;executors,&nbsp;all&nbsp;else&nbsp;equal.<br
/>{quote}<br />So,&nbsp;this&nbsp;does&nbsp;not&nbsp;seem&nbsp;to&nbsp;be&nbsp;what&nbsp;we're&nbsp;seeing.&nbsp;Here's&nbsp;output&nbsp;from&nbsp;our&nbsp;executors&nbsp;tab&nbsp;after&nbsp;running&nbsp;a&nbsp;while.&nbsp;Notice&nbsp;the&nbsp;executors&nbsp;with&nbsp;more&nbsp;RDD&nbsp;partitions&nbsp;have&nbsp;proportionally&nbsp;more&nbsp;tasks&nbsp;executed.<br
/><br /></span> <span class="diffaddedchars" style="background-color:#ddfade;">||</span>
<span class="diffcontext">Executor&nbsp;ID</span> <span class="diffaddedchars"
style="background-color:#ddfade;">||</span> <span class="diffcontext"> RDD&nbsp;Blocks</span>
<span class="diffaddedchars" style="background-color:#ddfade;">||</span> <span
class="diffcontext"> Storage&nbsp;Memory</span> <span class="diffaddedchars"
style="background-color:#ddfade;">||</span> <span class="diffcontext"> Disk&nbsp;Used</span>
<span class="diffaddedchars" style="background-color:#ddfade;">||</span> <span
class="diffcontext"> Active&nbsp;Tasks</span> <span class="diffaddedchars"
style="background-color:#ddfade;">||</span> <span class="diffcontext"> Failed&nbsp;Tasks</span>
<span class="diffaddedchars" style="background-color:#ddfade;">||</span> <span
class="diffcontext"> Complete&nbsp;Tasks</span> <span class="diffaddedchars"
style="background-color:#ddfade;">||</span> <span class="diffcontext"> Total&nbsp;Tasks<br
/>1</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 8</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 91.2&nbsp;MB&nbsp;/&nbsp;2.7&nbsp;GB</span>
<span class="diffaddedchars" style="background-color:#ddfade;">|</span> <span
class="diffcontext"> 0.0&nbsp;B</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 0</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 0</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 440804</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 440804<br /></span> <span class="diffremovedchars"
style="background-color:#ffe7e7;text-decoration:line-through;"> <br /></span>
<span class="diffcontext">10</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 30</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 342.0&nbsp;MB&nbsp;/&nbsp;2.7&nbsp;GB</span>
<span class="diffaddedchars" style="background-color:#ddfade;">|</span> <span
class="diffcontext"> 0.0&nbsp;B</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 0</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 0</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 722964</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 722964<br /></span> <span class="diffremovedchars"
style="background-color:#ffe7e7;text-decoration:line-through;"> <br /></span>
<span class="diffcontext">11</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 8</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 91.2&nbsp;MB&nbsp;/&nbsp;2.7&nbsp;GB</span>
<span class="diffaddedchars" style="background-color:#ddfade;">|</span> <span
class="diffcontext"> 0.0&nbsp;B</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 0</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 0</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 444454</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 444454<br /></span> <span class="diffremovedchars"
style="background-color:#ffe7e7;text-decoration:line-through;"> <br /></span>
<span class="diffcontext">12</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 27</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 307.8&nbsp;MB&nbsp;/&nbsp;2.7&nbsp;GB</span>
<span class="diffaddedchars" style="background-color:#ddfade;">|</span> <span
class="diffcontext"> 0.0&nbsp;B</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 0</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 0</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 780879</span> <span class="diffaddedchars" style="background-color:#ddfade;">|</span>
<span class="diffcontext"> 780879<br /></span> <span class="diffaddedchars"
style="background-color:#ddfade;"><br />With&nbsp;RDD&nbsp;partitions&nbsp;cached,&nbsp;wouldn't&nbsp;the&nbsp;scheduler&nbsp;prefer&nbsp;to&nbsp;run&nbsp;the&nbsp;task&nbsp;on&nbsp;the&nbsp;executor&nbsp;containing&nbsp;the&nbsp;blocks?<br
/></span> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand " style="padding:
0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc;
border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff"> 
                                <table id="actions-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule:
exactly; mso-text-raise: 1px"> 
                                    <tr> 
                                        <td id="actions-pattern-container" valign="middle"
style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align:
middle; padding-left: 0"> 
                                            <table align="left" style="border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
                                                <tr> 
                                                    <td class="actions-pattern-action-icon-container"
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; vertical-align:
middle"> <a href="https://issues.apache.org/jira/browse/SPARK-19371#add-comment" target="_blank"
title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image"
src="cid:jira-generated-image-static-comment-icon-12ad040b-b567-41d6-80f7-4552f625860a" alt="Add
Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"
/> </a> 
                                                    </td> 
                                                    <td class="actions-pattern-action-text-container"
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left:
5px"> <a href="https://issues.apache.org/jira/browse/SPARK-19371#add-comment" target="_blank"
title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a>

                                                    </td> 
                                                </tr> 
                                            </table> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <!-- there needs to be content in the cell for it to render in
some clients --> 
                        <tr> 
                            <td class="email-content-rounded-bottom mobile-expand" style="padding:
0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 5px; line-height:
5px; background-color: #fff; border-top: 0; border-left: 1px solid #ccc; border-bottom: 1px
solid #ccc; border-right: 1px solid #ccc; border-bottom-right-radius: 5px; border-bottom-left-radius:
5px; mso-line-height-rule: exactly">
                                &nbsp;
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
            <tr> 
                <td id="footer-pattern" style="padding: 0px; border-collapse: collapse;
padding: 12px 20px"> 
                    <table id="footer-pattern-container" cellspacing="0" cellpadding="0"
border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">

                        <tr> 
                            <td id="footer-pattern-text" class="mobile-resize-text" width="100%"
style="padding: 0px; border-collapse: collapse; color: #999; font-size: 12px; line-height:
18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px">
                                 This message was sent by Atlassian JIRA <span id="footer-build-information">(v6.3.15#6346-<span
title="dbc023dd75cecacf443c4b235f66124b15f5c5fe" data-commit-id="dbc023dd75cecacf443c4b235f66124b15f5c5fe}">sha1:dbc023d</span>)</span>

                            </td> 
                            <td id="footer-pattern-logo-desktop-container" valign="top"
style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">

                                <table style="border-collapse: collapse; mso-table-lspace:
0pt; mso-table-rspace: 0pt"> 
                                    <tr> 
                                        <td id="footer-pattern-logo-desktop-padding" style="padding:
0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop"
src="cid:jira-generated-image-static-footer-desktop-logo-9fe85ab1-0bef-483e-ab06-d3bea9d22aa4"
alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" />

                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
        </table>   
    </body>
</html>
Mime
View raw message