Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Mon, 6 Aug 2012 01:09:02 +0000 (UTC)
From: "Show You (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <1376498549.15324.1344215342414.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Created] (HIVE-3335) Thousand of CLOSE_WAIT socket when we
 using SymbolicInputFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Show You created HIVE-3335:
------------------------------

             Summary: Thousand of CLOSE_WAIT socket when we using SymbolicInputFormat
                 Key: HIVE-3335
                 URL: https://issues.apache.org/jira/browse/HIVE-3335
             Project: Hive
          Issue Type: Bug
          Components: Clients
    Affects Versions: 0.8.1
         Environment:  CentOS 5.8 x64
 CDH3u4
   hadoop-0.20-0.20.2+923.256-1
   hadoop-0.20-{namenode,secondarynamenode,jobtracker,tasktracker,datanode}-0.20.2+923.256-1
   hadoop-0.20-conf-pseudo-0.20.2+923.256-1(but same error was
occurred on not pseudo env)
 apache hive-0.8.1(but same error was occurred on hive 0.9)
            Reporter: Show You


Procedure for reproduction:
 1. Set up hadoop
 2. Prepare data file and link.txt:
    data:
      $ hadoop fs -cat /path/to/data/2012-07-01/20120701.csv
      1, 20120701 00:00:00
      2, 20120701 00:00:01
      3, 20120701 01:12:45
    link.txt
      $ cat link.txt
       /path/to/data/2012-07-01//*

 2. On hive, create table like below:
   CREATE TABLE user_logs(id INT, created_at STRING)
   row format delimited fields terminated by ',' lines terminated by '\n'
   stored as inputformat 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
   outputformat 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';

 3. Put link.txt to /user/hive/warehouse/user_logs
   $ sudo -u hdfs hadoop fs -put link.txt  /user/hive/warehouse/user_logs

 4. Open another session(A session), and watch socket,
   $ netstat -a | grep CLOSE_WAIT
    tcp        1      0 localhost:48121             localhost:50010
         CLOSE_WAIT
    tcp        1      0 localhost:48124             localhost:50010
         CLOSE_WAIT
   $

 5. Return to hive session, execute this,
   $ select * from user_logs;

 6. Return to A session, watch socket again,
   $ netstat -a | grep CLOSE_WAIT
   tcp        1      0 localhost:48121             localhost:50010
        CLOSE_WAIT
   tcp        1      0 localhost:48124             localhost:50010
        CLOSE_WAIT
   tcp        1      0 localhost:48166             localhost:50010
        CLOSE_WAIT

 If you makes any partitions, you'll watch unclosed socket whose count
equals partitions by once.


I think that this problem maybe is caused by this point:
  At https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/SymbolicInputFormat.java,
  line 66. BufferedReader was opened, but it doesn't closed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira