hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthik K <oss....@gmail.com>
Subject Re: Building offline
Date Fri, 19 Mar 2010 07:14:03 GMT
To add to chaitanya's suggestions earlier - you can set the version number
in libraries.properties / ivy.xml to be something else (for the patched
version you are working on, say 0.22.1 or 0.23.0 ) and publish using the
'ant mvn-install'  task.

Then in mapreduce's ivy.xml - just change the libraries.properties to the
new version that you had put and build ,to include your new version. Let us
know how it goes ( no need to go , --offline etc. ).




On Thu, Mar 18, 2010 at 7:54 PM, Rodrigo Schmidt <rodrigo@facebook.com>wrote:

>
> Ant clean is not erasing ~/.ivy2/cache, so I have to explicitly remove the
> files if I want to replace the jar for a newly compiled one. This doesn't
> sound right, though.
>
> Cheers,
> Rodrigo
>
> On 3/18/10 12:54 AM, "chaitanya krishna" <chaitanyavv.iiith@gmail.com>
> wrote:
>
> > Hi Rodrigo,
> >
> >   AFAIK,  I dont find any other way of cross-project debugging as of now.
> > Also, -Doffline=true is not working (a bug). Once this works, I think we
> can
> > work around by putting hadoop-core.jar in
> > mapred-trunk/build/ivy/lib/Hadoop/common (not sure though).
> >
> >  Also, I think doing an ant clean before running ant test
> > -Dresolvers=internal might be the actual solution instead of deleting
> > hadoop-core.jar in ~/.ivy2/cache, because the cached jar might be the one
> in
> > build/ivy/lib/Hadoop/common.
> >
> > Regards,
> > Chaitanya.
> >
> > On Thu, Mar 18, 2010 at 1:01 PM, Rodrigo Schmidt <rodrigo@facebook.com
> >wrote:
> >
> >>
> >> Thanks Chaitanya!
> >>
> >> Didn't work at first. I tried many times in different ways, connected to
> >> the
> >> Internet and not.
> >>
> >> When I was about to give up I realized ivy might be caching some stuff,
> >> which turned out to be true. I had to delete the hadoop-core.jar file
> >> inside
> >> ~/ivy2/cache for it to work.
> >>
> >> This whole ivy2 and maven dependencies look too complicated for offline
> use
> >> and cross-project debugging. Isn't there an easier way to do things?
> >>
> >> Cheers,
> >> Rodrigo
> >>
> >>
> >> On 3/17/10 11:27 PM, "chaitanya krishna" <chaitanyavv.iiith@gmail.com>
> >> wrote:
> >>
> >>> In case you want to use hadoop-core.jar containing your modifications
> in
> >>> common, i think the following works:
> >>>
> >>> 1. ant mvn-install in common project. This will create hadoop-core.jar
> >> and
> >>> installs it in the local m2 repository.
> >>> 2. ant test -Dresolvers=internal in mapreduce project. This will make
> >> sure
> >>> that the hadoop-core.jar is taken from local m2 repository.
> >>>
> >>> Hope this helps.
> >>>
> >>> -Chaitanya.
> >>>
> >>> On Thu, Mar 18, 2010 at 11:53 AM, Rodrigo Schmidt <
> rodrigo@facebook.com
> >>> wrote:
> >>>
> >>>>
> >>>> I tried that but my logs still didn't show up. To test if it really
> >> worked,
> >>>> I disabled my Internet connection and tried it, which failed. Did I
> miss
> >>>> anything?
> >>>>
> >>>>
> >>>> Output below:
> >>>>
> >>>>
> >>>> $ant -Doffline=yes test
> >>>> Buildfile: build.xml
> >>>>
> >>>> ivy-download:
> >>>>
> >>>> ivy-init-dirs:
> >>>>
> >>>> ivy-probe-antlib:
> >>>>
> >>>> ivy-init-antlib:
> >>>>
> >>>> ivy-init:
> >>>> [ivy:configure] :: Ivy 2.0.0-rc2 - 20081028224207 ::
> >>>> http://ant.apache.org/ivy/ ::
> >>>> :: loading settings :: file =
> >>>> /Users/rschmidt/workspace/hadoop-mapreduce-trunk/ivy/ivysettings.xml
> >>>>
> >>>> ivy-resolve-common:
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve] You probably access the destination server through a
> proxy
> >>>> server that is not well configured.
> >>>> [ivy:resolve]
> >>>> [ivy:resolve] :: problems summary ::
> >>>> [ivy:resolve] :::: WARNINGS
> >>>> [ivy:resolve]     Host repository.apache.org not found.
> >>>> url=
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>>>
> >>
> hadoop/hadoop-core/0.22.0-SNAPSHOT/maven-metadata.xml<https://repository.apac
> >>>>
> >>
> he.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoop-core/0.22.0
> >>>> -SNAPSHOT/maven-metadata.xml>
> >>>> [ivy:resolve]     Host repository.apache.org not found.
> >>>> url=
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>>> hadoop/hadoop-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.pom<
> >> https://re
> >>>>
> >>
> pository.apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoo
> >>>> p-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]     Host repository.apache.org not found.
> >>>> url=
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>>> hadoop/hadoop-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.jar<
> >> https://re
> >>>>
> >>
> pository.apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoo
> >>>> p-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]     Host repo1.maven.org not found.
> >>>> url=
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/0.22.0-SNAPS
> >>>> HOT/hadoop-core-0.22.0-SNAPSHOT.pom<
> >> http://repo1.maven.org/maven2/org/apache/
> >>>> hadoop/hadoop-core/0.22.0-SNAPS%0AHOT/hadoop-core-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]     Host repo1.maven.org not found.
> >>>> url=
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/0.22.0-SNAPS
> >>>> HOT/hadoop-core-0.22.0-SNAPSHOT.jar<
> >> http://repo1.maven.org/maven2/org/apache/
> >>>> hadoop/hadoop-core/0.22.0-SNAPS%0AHOT/hadoop-core-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]         module not found:
> >>>> org.apache.hadoop#hadoop-core;0.22.0-SNAPSHOT
> >>>> [ivy:resolve]     ==== apache-snapshot: tried
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >>>> op/hadoop-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.pom<
> >> https://reposi
> >>>>
> >>
> tory.apache.org/content/repositories/snapshots/org/apache/hado%0Aop/hadoop-co
> >>>> re/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]       -- artifact
> >>>> org.apache.hadoop#hadoop-core;0.22.0-SNAPSHOT!hadoop-core.jar:
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >>>> op/hadoop-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.jar<
> >> https://reposi
> >>>>
> >>
> tory.apache.org/content/repositories/snapshots/org/apache/hado%0Aop/hadoop-co
> >>>> re/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]     ==== maven2: tried
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/0.22.0-SNAPSHOT/
> >>>> hadoop-core-0.22.0-SNAPSHOT.pom<
> >> http://repo1.maven.org/maven2/org/apache/hado
> >>>> op/hadoop-core/0.22.0-SNAPSHOT/%0Ahadoop-core-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]       -- artifact
> >>>> org.apache.hadoop#hadoop-core;0.22.0-SNAPSHOT!hadoop-core.jar:
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/0.22.0-SNAPSHOT/
> >>>> hadoop-core-0.22.0-SNAPSHOT.jar<
> >> http://repo1.maven.org/maven2/org/apache/hado
> >>>> op/hadoop-core/0.22.0-SNAPSHOT/%0Ahadoop-core-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]     Host repository.apache.org not found.
> >>>> url=
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>>> hadoop/hadoop-core-test/0.22.0-SNAPSHOT/maven-metadata.xml<
> >> https://repository
> >>>> .
> >>
> apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoop-core-t
> >>>> est/0.22.0-SNAPSHOT/maven-metadata.xml>
> >>>> [ivy:resolve]     Host repository.apache.org not found.
> >>>> url=
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>>>
> >>>>
> >>
> hadoop/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom<
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/%0Aha
> >>>>
> >>
> doop/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]     Host repository.apache.org not found.
> >>>> url=
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>>>
> >>>>
> >>
> hadoop/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar<
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/%0Aha
> >>>>
> >>
> doop/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]     Host repo1.maven.org not found.
> >>>> url=
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core-test/0.22.0-
> >>>> SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom<
> >> http://repo1.maven.org/maven2/o
> >>>>
> >>
> rg/apache/hadoop/hadoop-core-test/0.22.0-%0ASNAPSHOT/hadoop-core-test-0.22.0-
> >>>> SNAPSHOT.pom>
> >>>> [ivy:resolve]     Host repo1.maven.org not found.
> >>>> url=
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core-test/0.22.0-
> >>>> SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar<
> >> http://repo1.maven.org/maven2/o
> >>>>
> >>
> rg/apache/hadoop/hadoop-core-test/0.22.0-%0ASNAPSHOT/hadoop-core-test-0.22.0-
> >>>> SNAPSHOT.jar>
> >>>> [ivy:resolve]         module not found:
> >>>> org.apache.hadoop#hadoop-core-test;0.22.0-SNAPSHOT
> >>>> [ivy:resolve]     ==== apache-snapshot: tried
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >>>>
> >>
> op/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom<http
> >>>> s://
> >>
> repository.apache.org/content/repositories/snapshots/org/apache/hado%0Aop
> >>>>
> /hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]       -- artifact
> >>>>
> org.apache.hadoop#hadoop-core-test;0.22.0-SNAPSHOT!hadoop-core-test.jar:
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >>>>
> >>
> op/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar<http
> >>>> s://
> >>
> repository.apache.org/content/repositories/snapshots/org/apache/hado%0Aop
> >>>>
> /hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]     ==== maven2: tried
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core-test/0.22.0-SNAP
> >>>> SHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom<
> >> http://repo1.maven.org/maven2/org/a
> >>>>
> >>
> pache/hadoop/hadoop-core-test/0.22.0-SNAP%0ASHOT/hadoop-core-test-0.22.0-SNAP
> >>>> SHOT.pom>
> >>>> [ivy:resolve]       -- artifact
> >>>>
> org.apache.hadoop#hadoop-core-test;0.22.0-SNAPSHOT!hadoop-core-test.jar:
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core-test/0.22.0-SNAP
> >>>> SHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar<
> >> http://repo1.maven.org/maven2/org/a
> >>>>
> >>
> pache/hadoop/hadoop-core-test/0.22.0-SNAP%0ASHOT/hadoop-core-test-0.22.0-SNAP
> >>>> SHOT.jar>
> >>>> [ivy:resolve]     Host repository.apache.org not found.
> >>>> url=
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>>>
> >>
> hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/maven-metadata.xml<https://repository.apac
> >>>>
> >>
> he.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoop-hdfs/0.22.0
> >>>> -SNAPSHOT/maven-metadata.xml>
> >>>> [ivy:resolve]     Host repository.apache.org not found.
> >>>> url=
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>>> hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom<
> >> https://re
> >>>>
> >>
> pository.apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoo
> >>>> p-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]     Host repository.apache.org not found.
> >>>> url=
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>>> hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar<
> >> https://re
> >>>>
> >>
> pository.apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoo
> >>>> p-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]     Host repo1.maven.org not found.
> >>>> url=
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPS
> >>>> HOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom<
> >> http://repo1.maven.org/maven2/org/apache/
> >>>> hadoop/hadoop-hdfs/0.22.0-SNAPS%0AHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]     Host repo1.maven.org not found.
> >>>> url=
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPS
> >>>> HOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar<
> >> http://repo1.maven.org/maven2/org/apache/
> >>>> hadoop/hadoop-hdfs/0.22.0-SNAPS%0AHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]         module not found:
> >>>> org.apache.hadoop#hadoop-hdfs;0.22.0-SNAPSHOT
> >>>> [ivy:resolve]     ==== apache-snapshot: tried
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >>>> op/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom<
> >> https://reposi
> >>>>
> >>
> tory.apache.org/content/repositories/snapshots/org/apache/hado%0Aop/hadoop-hd
> >>>> fs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]       -- artifact
> >>>> org.apache.hadoop#hadoop-hdfs;0.22.0-SNAPSHOT!hadoop-hdfs.jar:
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >>>> op/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar<
> >> https://reposi
> >>>>
> >>
> tory.apache.org/content/repositories/snapshots/org/apache/hado%0Aop/hadoop-hd
> >>>> fs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]     ==== maven2: tried
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/
> >>>> hadoop-hdfs-0.22.0-SNAPSHOT.pom<
> >> http://repo1.maven.org/maven2/org/apache/hado
> >>>> op/hadoop-hdfs/0.22.0-SNAPSHOT/%0Ahadoop-hdfs-0.22.0-SNAPSHOT.pom>
> >>>> [ivy:resolve]       -- artifact
> >>>> org.apache.hadoop#hadoop-hdfs;0.22.0-SNAPSHOT!hadoop-hdfs.jar:
> >>>> [ivy:resolve]
> >>>>
> >>>>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/
> >>>> hadoop-hdfs-0.22.0-SNAPSHOT.jar<
> >> http://repo1.maven.org/maven2/org/apache/hado
> >>>> op/hadoop-hdfs/0.22.0-SNAPSHOT/%0Ahadoop-hdfs-0.22.0-SNAPSHOT.jar>
> >>>> [ivy:resolve]         ::::::::::::::::::::::::::::::::::::::::::::::
> >>>> [ivy:resolve]         ::          UNRESOLVED DEPENDENCIES         ::
> >>>> [ivy:resolve]         ::::::::::::::::::::::::::::::::::::::::::::::
> >>>> [ivy:resolve]         ::
> org.apache.hadoop#hadoop-core;0.22.0-SNAPSHOT:
> >> not
> >>>> found
> >>>> [ivy:resolve]         ::
> >>>> org.apache.hadoop#hadoop-core-test;0.22.0-SNAPSHOT:
> >>>> not found
> >>>> [ivy:resolve]         ::
> org.apache.hadoop#hadoop-hdfs;0.22.0-SNAPSHOT:
> >> not
> >>>> found
> >>>> [ivy:resolve]         ::::::::::::::::::::::::::::::::::::::::::::::
> >>>> [ivy:resolve]
> >>>> [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> >>>>
> >>>>
> >>>>
> >>>> On 3/17/10 11:14 PM, "Amar Kamat" <amarrk@yahoo-inc.com> wrote:
> >>>>
> >>>>> Use -Doffline=yes switch while running the tests.
> >>>>> Amar
> >>>>>
> >>>>> On 3/18/10 11:32 AM, "Rodrigo Schmidt" <rodrigo@facebook.com>
wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> How do I disable the ivy download to run unit tests? I need to
> replace
> >>>>> hadoop-core.jar for one I created with some extra logging to debug
a
> >> unit
> >>>> test
> >>>>> for mapreduce.
> >>>>>
> >>>>> Right now I'm just erasing some lines from build.xml, but I'm quite
> >> sure
> >>>> there
> >>>>> is an easier way to do that.
> >>>>>
> >>>>> Thanks,
> >>>>> Rodrigo
> >>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message