spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Assudani (JIRA)" <j...@apache.org>
Subject [jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme
Date Mon, 30 Jan 2017 22:24:44 GMT
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml"> 
    <head> 
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 
        <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0"
/> <base href="https://issues.apache.org/jira" /> 
        <title>Message Title</title> 
    </head> 
    <body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: 14px;
line-height: 1.429"> 
        <table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
            <!-- header here --> 
            <tr> 
                <td id="header-pattern-container" style="padding: 0px; border-collapse:
collapse; padding: 10px 20px"> 
                    <table id="header-pattern" cellspacing="0" cellpadding="0" border="0"
style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
                        <tr> 
                            <td id="header-avatar-image-container" valign="top" style="padding:
0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px">
<img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-amit.assudani@gmail.com-5d6ada10-25d2-450b-970a-2ff63d7f5996"
height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" /> 
                            </td> 
                            <td id="header-text-container" valign="middle" style="padding:
0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover"
rel="amit.assudani@gmail.com" id="email_amit.assudani@gmail.com" href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=amit.assudani%40gmail.com"
style="color:#3b73af;; color: #3b73af; text-decoration: none">Amit Assudani</a> <strong>created</strong>
an issue 
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
            <tr> 
                <td id="email-content-container" style="padding: 0px; border-collapse:
collapse; padding: 0 20px"> 
                    <table id="email-content-table" cellspacing="0" cellpadding="0" border="0"
width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt;
border-spacing: 0; border-collapse: separate"> 
                        <tr> 
                            <!-- there needs to be content in the cell for it to render
in some clients --> 
                            <td class="email-content-rounded-top mobile-expand" style="padding:
0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 15px; background-color:
#fff; border-left: 1px solid #ccc; border-top: 1px solid #ccc; border-right: 1px solid #ccc;
border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px;
line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly">
                                &nbsp;
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand " style="padding:
0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc;
border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff"> 
                                <table class="page-title-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt"> 
                                    <tr> 
                                        <td class="page-title-pattern-first-line " style="padding:
0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; padding-top:
10px"> <a href="https://issues.apache.org/jira/browse/SPARK" style="color: #3b73af;
text-decoration: none">Spark</a> / <a href="https://issues.apache.org/jira/browse/SPARK-19407"
style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-static-bug-a5c9727e-2f3c-47aa-b170-e29ebd4f3d7a"
height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: text-bottom"
/></a> <a href="https://issues.apache.org/jira/browse/SPARK-19407" style="color:
#3b73af; text-decoration: none">SPARK-19407</a> 
                                        </td> 
                                    </tr> 
                                    <tr> 
                                        <td style="vertical-align: top;; padding: 0px;
border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule:
exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header"
style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise:
2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.apache.org/jira/browse/SPARK-19407"
style="color: #3b73af; text-decoration: none">defaultFS is used FileSystem.get instead
of getting it from uri scheme</a> </span> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand  wrapper-special-margin"
style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right:
1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color:
#fff; padding-top: 10px; padding-bottom: 5px"> 
                                <table class="keyvalue-table" style="border-collapse: collapse;
mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
                                    <tr> 
                                        <th style="color: #707070; font: normal 14px/20px
Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
                                            Issue Type:
                                        </th> 
                                        <td class="has-icon" style="padding: 0px; border-collapse:
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align:
top"> <img src="cid:jira-generated-image-static-bug-a5c9727e-2f3c-47aa-b170-e29ebd4f3d7a"
height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: text-bottom"
/> Bug 
                                        </td> 
                                    </tr> 
                                    <tr> 
                                        <th style="color: #707070; font: normal 14px/20px
Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
                                            Affects Versions:
                                        </th> 
                                        <td style="padding: 0px; border-collapse: collapse;
font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
                                             2.1.0 
                                        </td> 
                                    </tr> 
                                    <tr> 
                                        <th style="color: #707070; font: normal 14px/20px
Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
                                            Assignee:
                                        </th> 
                                        <td style="padding: 0px; border-collapse: collapse;
font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
                                             Unassigned 
                                        </td> 
                                    </tr> 
                                    <tr> 
                                        <th style="color: #707070; font: normal 14px/20px
Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
                                            Components:
                                        </th> 
                                        <td style="padding: 0px; border-collapse: collapse;
font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
                                             Structured Streaming 
                                        </td> 
                                    </tr> 
                                    <tr> 
                                        <th style="color: #707070; font: normal 14px/20px
Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
                                            Created:
                                        </th> 
                                        <td style="padding: 0px; border-collapse: collapse;
font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
                                             30/Jan/17 22:24 
                                        </td> 
                                    </tr> 
                                    <tr> 
                                        <th style="color: #707070; font: normal 14px/20px
Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
                                            Labels:
                                        </th> 
                                        <td style="padding: 0px; border-collapse: collapse;
font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
                                             streaming checkpoint filesystem 
                                        </td> 
                                    </tr> 
                                    <tr> 
                                        <th style="color: #707070; font: normal 14px/20px
Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
                                            Priority:
                                        </th> 
                                        <td class="has-icon" style="padding: 0px; border-collapse:
collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align:
top"> <img src="cid:jira-generated-image-static-major-7c1777d4-c94a-4a71-8c0d-d6874adfd1da"
height="16" width="16" border="0" align="absmiddle" alt="Major" style="vertical-align: text-bottom"
/> Major 
                                        </td> 
                                    </tr> 
                                    <tr> 
                                        <th style="color: #707070; font: normal 14px/20px
Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
                                            Reporter:
                                        </th> 
                                        <td style="padding: 0px; border-collapse: collapse;
font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
<a class="user-hover" rel="amit.assudani@gmail.com" id="email_amit.assudani@gmail.com"
href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=amit.assudani%40gmail.com"
style="color:#3b73af;; color: #3b73af; text-decoration: none">Amit Assudani</a> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand  issue-description-container"
style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right:
1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color:
#fff; padding-top: 5px; padding-bottom: 10px"> 
                                <table class="text-paragraph-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule:
exactly; mso-text-raise: 2px"> 
                                    <tr> 
                                        <td class="text-paragraph-pattern-container mobile-resize-text
" style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0"> 
                                            <p style="margin-top:0;margin-bottom:10px;;
margin: 10px 0 0 0">Caused by: java.lang.IllegalArgumentException: Wrong FS: s3a://**************/checkpoint/7b2231a3-d845-4740-bfa3-681850e5987f/metadata,
expected: <a href="file:///" class="external-link" rel="nofollow" style="color: #3b73af;
text-decoration: none">file:///</a><br /> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:649)<br
/> at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)<br
/> at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)<br
/> at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)<br
/> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)<br
/> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)<br
/> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)<br /> at org.apache.spark.sql.execution.streaming.StreamMetadata$.read(StreamMetadata.scala:51)<br
/> at org.apache.spark.sql.execution.streaming.StreamExecution.&lt;init&gt;(StreamExecution.scala:100)<br
/> at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:232)<br
/> at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:269)<br
/> at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:262)</p>

                                            <p style="margin-top:0;margin-bottom:10px;;
margin: 10px 0 0 0">Can easily replicate on spark standalone cluster by providing checkpoint
location uri scheme anything other than &quot;file://&quot; and not overriding in
config.</p> 
                                            <p style="margin-top:0;margin-bottom:10px;;
margin: 10px 0 0 0">WorkAround --conf spark.hadoop.fs.defaultFS=s3a://somebucket or set
it in sparkConf or spark-default.conf</p> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <tr> 
                            <td class="email-content-main mobile-expand " style="padding:
0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc;
border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff"> 
                                <table id="actions-pattern" cellspacing="0" cellpadding="0"
border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace:
0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule:
exactly; mso-text-raise: 1px"> 
                                    <tr> 
                                        <td id="actions-pattern-container" valign="middle"
style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align:
middle; padding-left: 0"> 
                                            <table align="left" style="border-collapse:
collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt"> 
                                                <tr> 
                                                    <td class="actions-pattern-action-icon-container"
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; vertical-align:
middle"> <a href="https://issues.apache.org/jira/browse/SPARK-19407#add-comment" target="_blank"
title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image"
src="cid:jira-generated-image-static-comment-icon-e079135c-2709-4672-94b9-daf71bb0095a" alt="Add
Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"
/> </a> 
                                                    </td> 
                                                    <td class="actions-pattern-action-text-container"
style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size:
14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left:
5px"> <a href="https://issues.apache.org/jira/browse/SPARK-19407#add-comment" target="_blank"
title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a>

                                                    </td> 
                                                </tr> 
                                            </table> 
                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                        <!-- there needs to be content in the cell for it to render in
some clients --> 
                        <tr> 
                            <td class="email-content-rounded-bottom mobile-expand" style="padding:
0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 5px; line-height:
5px; background-color: #fff; border-top: 0; border-left: 1px solid #ccc; border-bottom: 1px
solid #ccc; border-right: 1px solid #ccc; border-bottom-right-radius: 5px; border-bottom-left-radius:
5px; mso-line-height-rule: exactly">
                                &nbsp;
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
            <tr> 
                <td id="footer-pattern" style="padding: 0px; border-collapse: collapse;
padding: 12px 20px"> 
                    <table id="footer-pattern-container" cellspacing="0" cellpadding="0"
border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">

                        <tr> 
                            <td id="footer-pattern-text" class="mobile-resize-text" width="100%"
style="padding: 0px; border-collapse: collapse; color: #999; font-size: 12px; line-height:
18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px">
                                 This message was sent by Atlassian JIRA <span id="footer-build-information">(v6.3.15#6346-<span
title="dbc023dd75cecacf443c4b235f66124b15f5c5fe" data-commit-id="dbc023dd75cecacf443c4b235f66124b15f5c5fe}">sha1:dbc023d</span>)</span>

                            </td> 
                            <td id="footer-pattern-logo-desktop-container" valign="top"
style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">

                                <table style="border-collapse: collapse; mso-table-lspace:
0pt; mso-table-rspace: 0pt"> 
                                    <tr> 
                                        <td id="footer-pattern-logo-desktop-padding" style="padding:
0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop"
src="cid:jira-generated-image-static-footer-desktop-logo-c2cc743d-ad0c-4854-95a6-fdf26e8cb647"
alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" />

                                        </td> 
                                    </tr> 
                                </table> 
                            </td> 
                        </tr> 
                    </table> 
                </td> 
            </tr> 
        </table>   
    </body>
</html>
Mime
View raw message