hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Damien Carol (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-8071) hive shell tries to write hive-exec.jar for each run
Date Mon, 15 Sep 2014 08:56:36 GMT

     [ https://issues.apache.org/jira/browse/HIVE-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Damien Carol updated HIVE-8071:
-------------------------------
    Description: 
For every run of the hive CLI there is a delay for the shell startup
{noformat}
14/07/31 23:07:19 INFO Configuration.deprecation: fs.default.name is deprecated. Instead,
use fs.defaultFS
14/07/31 23:07:19 INFO tez.DagUtils: Hive jar directory is hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/
14/07/31 23:07:19 INFO tez.DagUtils: Localizing resource because it does not exist: file:/home/gopal/tez-autobuild/dist/hive/lib/hive-exec-0.14.0-SNAPSHOT.jar
to dest: 
hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/hive-exec-0.14.0-SNAPSHOTde1f82f0b5561d3db9e3080dfb2897210a3bda4ca5e7b14e881e381115837fd8.
jar
14/07/31 23:07:19 INFO tez.DagUtils: Looks like another thread is writing the same file will
wait.
14/07/31 23:07:19 INFO tez.DagUtils: Number of wait attempts: 5. Wait interval: 5000
14/07/31 23:07:19 INFO tez.DagUtils: Resource modification time: 1406870512963
14/07/31 23:07:20 INFO tez.TezSessionState: Opening new Tez Session (id: 02d6b558-44cc-4182-b2f2-6a37ffdd25d2,
scratch dir: hdfs://mac-10:8020/tmp/hive-gopal/_tez_session_dir/02d6b558-44cc-4182-b2f2-6a37ffdd25d2)
{noformat}

Traced this to a method which does PRIVATE LRs - this is marked as PRIVATE even if it is from
a common install dir.
{code}
 public LocalResource localizeResource(Path src, Path dest, Configuration conf)
    throws IOException {
....
    return createLocalResource(destFS, dest, LocalResourceType.FILE,
        LocalResourceVisibility.PRIVATE);
  }
{code}

  was:


For every run of the hive CLI there is a delay for the shell startup

14/07/31 23:07:19 INFO Configuration.deprecation: fs.default.name is deprecated. Instead,
use fs.defaultFS
14/07/31 23:07:19 INFO tez.DagUtils: Hive jar directory is hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/
14/07/31 23:07:19 INFO tez.DagUtils: Localizing resource because it does not exist: file:/home/gopal/tez-autobuild/dist/hive/lib/hive-exec-0.14.0-SNAPSHOT.jar
to dest: 
hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/hive-exec-0.14.0-SNAPSHOTde1f82f0b5561d3db9e3080dfb2897210a3bda4ca5e7b14e881e381115837fd8.
jar
14/07/31 23:07:19 INFO tez.DagUtils: Looks like another thread is writing the same file will
wait.
14/07/31 23:07:19 INFO tez.DagUtils: Number of wait attempts: 5. Wait interval: 5000
14/07/31 23:07:19 INFO tez.DagUtils: Resource modification time: 1406870512963
14/07/31 23:07:20 INFO tez.TezSessionState: Opening new Tez Session (id: 02d6b558-44cc-4182-b2f2-6a37ffdd25d2,
scratch dir: hdfs://mac-10:8020/tmp/hive-gopal/_tez_session_dir/02d6b558-44cc-4182-b2f2-6a37ffdd25d2)

Traced this to a method which does PRIVATE LRs - this is marked as PRIVATE even if it is from
a common install dir.
{code}
 public LocalResource localizeResource(Path src, Path dest, Configuration conf)
    throws IOException {
....
    return createLocalResource(destFS, dest, LocalResourceType.FILE,
        LocalResourceVisibility.PRIVATE);
  }
{code}


> hive shell tries to write hive-exec.jar for each run
> ----------------------------------------------------
>
>                 Key: HIVE-8071
>                 URL: https://issues.apache.org/jira/browse/HIVE-8071
>             Project: Hive
>          Issue Type: Bug
>          Components: Tez
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>         Attachments: HIVE-8071.1.patch
>
>
> For every run of the hive CLI there is a delay for the shell startup
> {noformat}
> 14/07/31 23:07:19 INFO Configuration.deprecation: fs.default.name is deprecated. Instead,
use fs.defaultFS
> 14/07/31 23:07:19 INFO tez.DagUtils: Hive jar directory is hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/
> 14/07/31 23:07:19 INFO tez.DagUtils: Localizing resource because it does not exist: file:/home/gopal/tez-autobuild/dist/hive/lib/hive-exec-0.14.0-SNAPSHOT.jar
to dest: 
> hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/hive-exec-0.14.0-SNAPSHOTde1f82f0b5561d3db9e3080dfb2897210a3bda4ca5e7b14e881e381115837fd8.
> jar
> 14/07/31 23:07:19 INFO tez.DagUtils: Looks like another thread is writing the same file
will wait.
> 14/07/31 23:07:19 INFO tez.DagUtils: Number of wait attempts: 5. Wait interval: 5000
> 14/07/31 23:07:19 INFO tez.DagUtils: Resource modification time: 1406870512963
> 14/07/31 23:07:20 INFO tez.TezSessionState: Opening new Tez Session (id: 02d6b558-44cc-4182-b2f2-6a37ffdd25d2,
scratch dir: hdfs://mac-10:8020/tmp/hive-gopal/_tez_session_dir/02d6b558-44cc-4182-b2f2-6a37ffdd25d2)
> {noformat}
> Traced this to a method which does PRIVATE LRs - this is marked as PRIVATE even if it
is from a common install dir.
> {code}
>  public LocalResource localizeResource(Path src, Path dest, Configuration conf)
>     throws IOException {
> ....
>     return createLocalResource(destFS, dest, LocalResourceType.FILE,
>         LocalResourceVisibility.PRIVATE);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message