hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14770) S3A http connection in s3a driver not reuse in Spark application
Date Mon, 14 Aug 2017 10:59:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125536#comment-16125536
] 

Yonger commented on HADOOP-14770:
---------------------------------

Thanks Steve, the application running on Hadoop 2.7.3 and against ORC file format. I will
upgrade to Hadoop 2.8.0 to verify.

> S3A http connection in s3a driver not reuse in Spark application
> ----------------------------------------------------------------
>
>                 Key: HADOOP-14770
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14770
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.7.3
>            Reporter: Yonger
>            Assignee: Yonger
>
> I print out connection stats every 2 s when running Spark application against s3-compatible
storage:
> {code}
> ESTAB      0      0         ::ffff:10.0.2.36:44446                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44454                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44374                ::ffff:10.0.2.254:80
                
> ESTAB      159724 0         ::ffff:10.0.2.36:44436                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44448                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44338                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44438                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44414                ::ffff:10.0.2.254:80
                
> ESTAB      0      480       ::ffff:10.0.2.36:44450                ::ffff:10.0.2.254:80
                 timer:(on,170ms,0)
> ESTAB      0      0         ::ffff:10.0.2.36:44442                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44390                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44326                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44452                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44394                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44444                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44456                ::ffff:10.0.2.254:80
                
> ======================
> ESTAB      0      0         ::ffff:10.0.2.36:44508                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44476                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44524                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44374                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44500                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44504                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44512                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44506                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44464                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44518                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44510                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44442                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44526                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44472                ::ffff:10.0.2.254:80
                
> ESTAB      0      0         ::ffff:10.0.2.36:44466                ::ffff:10.0.2.254:80

> {code}
> the connection in the above of "=" and below were changed all the time. But this haven't
seen in MR application. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message