hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Show You (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-3335) Thousand of CLOSE_WAIT socket when we using SymbolicInputFormat
Date Sun, 21 Oct 2012 09:18:12 GMT

    [ https://issues.apache.org/jira/browse/HIVE-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480933#comment-13480933
] 

Show You commented on HIVE-3335:
--------------------------------

I'm sorry to too late. I attach a patch for this issue.
                
> Thousand of CLOSE_WAIT socket when we using SymbolicInputFormat
> ---------------------------------------------------------------
>
>                 Key: HIVE-3335
>                 URL: https://issues.apache.org/jira/browse/HIVE-3335
>             Project: Hive
>          Issue Type: Bug
>          Components: Clients
>    Affects Versions: 0.8.1
>         Environment:  CentOS 5.8 x64
>  CDH3u4
>    hadoop-0.20-0.20.2+923.256-1
>    hadoop-0.20-{namenode,secondarynamenode,jobtracker,tasktracker,datanode}-0.20.2+923.256-1
>    hadoop-0.20-conf-pseudo-0.20.2+923.256-1(but same error was
> occurred on not pseudo env)
>  apache hive-0.8.1(but same error was occurred on hive 0.9)
>            Reporter: Show You
>         Attachments: HIVE-3335.patch
>
>
> Procedure for reproduction:
>  1. Set up hadoop
>  2. Prepare data file and link.txt:
>     data:
>       $ hadoop fs -cat /path/to/data/2012-07-01/20120701.csv
>       1, 20120701 00:00:00
>       2, 20120701 00:00:01
>       3, 20120701 01:12:45
>     link.txt
>       $ cat link.txt
>        /path/to/data/2012-07-01//*
>  2. On hive, create table like below:
>    CREATE TABLE user_logs(id INT, created_at STRING)
>    row format delimited fields terminated by ',' lines terminated by '\n'
>    stored as inputformat 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
>    outputformat 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
>  3. Put link.txt to /user/hive/warehouse/user_logs
>    $ sudo -u hdfs hadoop fs -put link.txt  /user/hive/warehouse/user_logs
>  4. Open another session(A session), and watch socket,
>    $ netstat -a | grep CLOSE_WAIT
>     tcp        1      0 localhost:48121             localhost:50010
>          CLOSE_WAIT
>     tcp        1      0 localhost:48124             localhost:50010
>          CLOSE_WAIT
>    $
>  5. Return to hive session, execute this,
>    $ select * from user_logs;
>  6. Return to A session, watch socket again,
>    $ netstat -a | grep CLOSE_WAIT
>    tcp        1      0 localhost:48121             localhost:50010
>         CLOSE_WAIT
>    tcp        1      0 localhost:48124             localhost:50010
>         CLOSE_WAIT
>    tcp        1      0 localhost:48166             localhost:50010
>         CLOSE_WAIT
>  If you makes any partitions, you'll watch unclosed socket whose count
> equals partitions by once.
> I think that this problem maybe is caused by this point:
>   At https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/SymbolicInputFormat.java,
>   line 66. BufferedReader was opened, but it doesn't closed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message