hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasha Ovsankin <sa...@codebistro.com>
Subject Re: Building Hive - cannot resolve dependencies
Date Thu, 12 Nov 2009 23:25:19 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html; charset=windows-1252"
 http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<blockquote type="cite">
  <blockquote class="gmail_quote"
 style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:
1ex;">Todd,
Zheng --
    <div><br>
    </div>
    <div>The build process relies on downloading 4 different (!) hadoop
versions from <a moz-do-not-send="true"
 href="http://archive.apache.org/dist/hadoop/core/hadoop-0.20.0/"
 target="_blank">http://archive.apache.org/dist/hadoop/core/hadoop-0.20.0/</a>
, a total of 160MB and fails if something goes wrong.</div>
    <div><br>
    </div>
    <div>This doesn't make a lot of sense -- why do you need 4 versions?</div>
  </blockquote>
  <br>
Hive
includes shims so that it can run against all of these versions. In
truth it really only needs the jars from these releases, but since
Hadoop isn't currently publishing its jars to a maven repo or anything,
we have to get the tarballs. I agree that this is a little annoying,
but once you've done it once it stays cached and you shouldn't have to
worry about it.</blockquote>
OK, I understand now. If all you need is jar files then the easiest
would be to create a repository that contains these files. <br>
<br>
Since I am not very well versed with Ivy, I have created a Maven
repository as a project on Google Code and put all 4 JAR files there. I
guess with a bit of luck one can point Ivy to this repository instead
of <a moz-do-not-send="true" href="http://archive.apache.org"
 target="_blank">http://archive.apache.org</a> and save the whole
download mess. This repository contains only the JAR files and will
download 20 times faster. The repo is located at:<br>
<blockquote><a class="moz-txt-link-freetext" href="http://hadoop-via-maven.googlecode.com/svn/trunk/repo/org/apache/hadoop/hadoop-core/">http://hadoop-via-maven.googlecode.com/svn/trunk/repo/org/apache/hadoop/hadoop-core/</a><br>
</blockquote>
I have tried to point Ivy to it by changing ivy/ivysettings.xml to :<br>
<blockquote>     &lt;artifact
pattern=<a class="moz-txt-link-rfc2396E" href="http://hadoop-via-maven.googlecode.com/svn/trunk/repo/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]">"http://hadoop-via-maven.googlecode.com/svn/trunk/repo/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"</a>/&gt;<br>
</blockquote>
, but this didn't work -- the build wants the complete tarballs. Guess
I would need to do more digging into the build process :-(<br>
<br>
The project is on <a class="moz-txt-link-freetext" href="http://code.google.com/p/hadoop-via-maven/">http://code.google.com/p/hadoop-via-maven/</a>.
I think
there are quite a few folks craving for this kind of repository.<br>
<blockquote type="cite">
  <blockquote class="gmail_quote"
 style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:
1ex;">
    <div><br>
    </div>
    <div>Also,
could you provide error message that explains how to manually download
and install the dependencies, not only "try again". Or point to the doc
that does.</div>
  </blockquote>
  <br>
This is a good idea. Patches welcome :)</blockquote>
This happens in build-common.xml, line 78 . Would love to help but I
need to learn a bit more of Ant and Ivy in order to do this...<br>
<blockquote type="cite">
  <blockquote class="gmail_quote"
 style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:
1ex;">
    <div><br>
    </div>
    <div>BTW there might be a problem with <a moz-do-not-send="true"
 href="http://archive.apache.org" target="_blank">http://archive.apache.org</a> server
-- it may be timing out connections which got me into an infinite loop
of "try again".</div>
    <div><br>
    </div>
  </blockquote>
  <br>
I haven't run into that before.
This problem with 0.19 is new (didn't happen when we committed this
built process). I've written up a sticky note on my monitor to take a
look when I have a moment.</blockquote>
For me, this consistently happened with all three 0.18 - 0.20. Most
likely the tarball size and/or download time has something to do with
this. I downloaded them with no problem to an EC2 machine though. Some
bad guy sitting on the line...<br>
<br>
<br>
On 11/12/09 12:49 PM, Todd Lipcon wrote:
<blockquote
 cite="mid:45f85f70911121249v6bf0b502kf890b1dec1339a35@mail.gmail.com"
 type="cite">On Thu, Nov 12, 2009 at 12:45 PM, Sasha Ovsankin <span
 dir="ltr">&lt;<a moz-do-not-send="true"
 href="mailto:sasha@codebistro.com">sasha@codebistro.com</a>&gt;</span>
wrote:<br>
  <div class="gmail_quote">
  <blockquote class="gmail_quote"
 style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:
1ex;">Todd,
Zheng --
    <div><br>
    </div>
    <div>The build process relies on downloading 4 different (!) hadoop
versions from <a moz-do-not-send="true"
 href="http://archive.apache.org/dist/hadoop/core/hadoop-0.20.0/"
 target="_blank">http://archive.apache.org/dist/hadoop/core/hadoop-0.20.0/</a>
, a total of 160MB and fails if something goes wrong.</div>
    <div><br>
    </div>
    <div>This doesn't make a lot of sense -- why do you need 4 versions?</div>
  </blockquote>
  <div><br>
Hive includes shims so that it can run against all of these versions.
In truth it really only needs the jars from these releases, but since
Hadoop isn't currently publishing its jars to a maven repo or anything,
we have to get the tarballs. I agree that this is a little annoying,
but once you've done it once it stays cached and you shouldn't have to
worry about it.<br>
 <br>
  </div>
  <blockquote class="gmail_quote"
 style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:
1ex;">
    <div><br>
    </div>
    <div>Also, could you provide error message that explains how to
manually download and install the dependencies, not only "try again".
Or point to the doc that does.</div>
  </blockquote>
  <div><br>
This is a good idea. Patches welcome :)<br>
 </div>
  <blockquote class="gmail_quote"
 style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:
1ex;">
    <div><br>
    </div>
    <div>BTW there might be a problem with <a moz-do-not-send="true"
 href="http://archive.apache.org" target="_blank">http://archive.apache.org</a> server
-- it may be timing out connections which got me into an infinite loop
of "try again".</div>
    <div><br>
    </div>
  </blockquote>
  <div><br>
I haven't run into that before. This problem with 0.19 is new (didn't
happen when we committed this built process). I've written up a sticky
note on my monitor to take a look when I have a moment.<br>
  <br>
-Todd<br>
 <br>
  </div>
  <blockquote class="gmail_quote"
 style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:
1ex;">
    <div>Thanks,</div>
    <div>-- Sasha</div>
    <div>
    <div class="h5">
    <div><br>
    <div class="gmail_quote">On Thu, Nov 12, 2009 at 2:30 AM, Zheng
Shao <span dir="ltr">&lt;<a moz-do-not-send="true"
 href="mailto:zshao9@gmail.com" target="_blank">zshao9@gmail.com</a>&gt;</span>
wrote:<br>
    <blockquote class="gmail_quote"
 style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:
1ex;">Hi
Todd,<br>
      <br>
I don't have a clear idea but it seems the ivy download code has a bug?<br>
People get the files by doing "wget" on the same machine and it
succeeds.<br>
      <br>
If you have time, maybe we can debug into ivy's download code?<br>
      <br>
Zheng<br>
      <div>
      <div><br>
On Thu, Nov 12, 2009 at 9:23 AM, Todd Lipcon &lt;<a
 moz-do-not-send="true" href="mailto:todd@cloudera.com" target="_blank">todd@cloudera.com</a>&gt;
wrote:<br>
&gt; Hey Zheng,<br>
&gt;<br>
&gt; What do we need to do to fix this? It seems to have bitten a
number of<br>
&gt; people by now.<br>
&gt;<br>
&gt; -Todd<br>
&gt;<br>
&gt; On Tue, Nov 10, 2009 at 3:50 PM, Zheng Shao &lt;<a
 moz-do-not-send="true" href="mailto:zshao9@gmail.com" target="_blank">zshao9@gmail.com</a>&gt;
wrote:<br>
&gt;&gt;<br>
&gt;&gt; I am forwarding an earlier email from the same mailing list by
search<br>
&gt;&gt; for "Downloaded file size doesn't match expected Content
Length":<br>
&gt;&gt;<br>
&gt;&gt; "<br>
&gt;&gt; Hi Rahul,<br>
&gt;&gt;<br>
&gt;&gt; Please follow these steps:<br>
&gt;&gt;<br>
&gt;&gt; 1) In your hive source directory run 'ant clean'.<br>
&gt;&gt; 2) remove the contents of ~/.ant/cache/hadoop/core/sources<br>
&gt;&gt; 3) Download the following files to
~/.ant/cache/hadoop/core/sources:<br>
&gt;&gt;<br>
&gt;&gt;      hadoop-0.17.2.1.tar.gz<br>
&gt;&gt;      hadoop-0.17.2.1.tar.gz.asc<br>
&gt;&gt;      hadoop-0.18.3.tar.gz<br>
&gt;&gt;      hadoop-0.18.3.tar.gz.asc<br>
&gt;&gt;<br>
&gt;&gt;      hadoop-0.19.0.tar.gz<br>
&gt;&gt;      hadoop-0.19.0.tar.gz.asc<br>
&gt;&gt;      hadoop-0.20.0.tar.gz<br>
&gt;&gt;      hadoop-0.20.0.tar.gz.asc<br>
&gt;&gt;<br>
&gt;&gt; 4) For each hadoop-xxx.tar.gz file, compute the sha1 checksum
using<br>
&gt;&gt; sha1sum, and verify that it matches the sha1 checksum in the<br>
&gt;&gt; corresponding .asc file.<br>
&gt;&gt;<br>
&gt;&gt; If it does not match then the file is corrupt and you need to
try<br>
&gt;&gt; downloading it again.<br>
&gt;&gt;<br>
&gt;&gt; 5) Try building Hive again following the instructions on the
wiki. You<br>
&gt;&gt; shouldn't have any problems if you verified the checksums.<br>
&gt;&gt;<br>
&gt;&gt; As an additional note, if you don't care about support for
Hadoop<br>
&gt;&gt; 0.17.2.1, or 0.18, etc, you can disable support for these
versions<br>
&gt;&gt; (and skip the download) by removing the references to these
versions<br>
&gt;&gt; that shims/build.xml and shims/ivy.xml<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;    Also I want to use hive on top of current working hadoop
cluster.<br>
&gt;&gt;    Please provide some inputs.<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; You need to set HADOOP_HOME and make sure that
HADOOP_HOME/conf (or<br>
&gt;&gt; HADOOP_CONF_DIR if you have this variable set) contains valid<br>
&gt;&gt; configuration files for your current working hadoop cluster.
See the<br>
&gt;&gt; following page for more information:<br>
&gt;&gt; <a moz-do-not-send="true"
 href="http://wiki.apache.org/hadoop/GettingStartedWithHadoop"
 target="_blank">http://wiki.apache.org/hadoop/GettingStartedWithHadoop</a><br>
&gt;&gt;<br>
&gt;&gt; Thanks.<br>
&gt;&gt;<br>
&gt;&gt; Carl<br>
&gt;&gt; "<br>
&gt;&gt;<br>
&gt;&gt; On Tue, Nov 10, 2009 at 3:39 PM, Sasha Ovsankin &lt;<a
 moz-do-not-send="true" href="mailto:sasha@codebistro.com"
 target="_blank">sasha@codebistro.com</a>&gt;<br>
&gt;&gt; wrote:<br>
&gt;&gt; &gt; Hello, all --<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; I am trying to build as per GettingStarted and am getting
this error<br>
&gt;&gt; &gt; from<br>
&gt;&gt; &gt; Ivy:<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; Downloaded file size doesn't match expected Content
Length for<br>
&gt;&gt; &gt; ...hadoop-0.19.0.tar.gz. Please retry.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; This error repeats 4 times for 4 different versions of
Hadoop and the<br>
&gt;&gt; &gt; build<br>
&gt;&gt; &gt; stops.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; I have tried on Mac and Windows with the same results.
How can I build<br>
&gt;&gt; &gt; Hive?<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; Thanks for your help,<br>
&gt;&gt; &gt; -- Sasha<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt; P.S. This is try#3 to get over the spam filter.<br>
&gt;&gt; &gt;<br>
&gt;&gt; &gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; --<br>
&gt;&gt; Yours,<br>
&gt;&gt; Zheng<br>
&gt;<br>
&gt;<br>
      <br>
      <br>
      <br>
      </div>
      </div>
--<br>
Yours,<br>
      <font color="#888888">Zheng<br>
      </font></blockquote>
    </div>
    <br>
    </div>
    </div>
    </div>
  </blockquote>
  </div>
  <br>
</blockquote>
</body>
</html>

Mime
View raw message