hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chaitanya krishna <chaitanyavv.ii...@gmail.com>
Subject Re: Building offline
Date Thu, 18 Mar 2010 07:54:01 GMT
Hi Rodrigo,

  AFAIK,  I dont find any other way of cross-project debugging as of now.
Also, -Doffline=true is not working (a bug). Once this works, I think we can
work around by putting hadoop-core.jar in
mapred-trunk/build/ivy/lib/Hadoop/common (not sure though).

 Also, I think doing an ant clean before running ant test
-Dresolvers=internal might be the actual solution instead of deleting
hadoop-core.jar in ~/.ivy2/cache, because the cached jar might be the one in
build/ivy/lib/Hadoop/common.

Regards,
Chaitanya.

On Thu, Mar 18, 2010 at 1:01 PM, Rodrigo Schmidt <rodrigo@facebook.com>wrote:

>
> Thanks Chaitanya!
>
> Didn't work at first. I tried many times in different ways, connected to
> the
> Internet and not.
>
> When I was about to give up I realized ivy might be caching some stuff,
> which turned out to be true. I had to delete the hadoop-core.jar file
> inside
> ~/ivy2/cache for it to work.
>
> This whole ivy2 and maven dependencies look too complicated for offline use
> and cross-project debugging. Isn't there an easier way to do things?
>
> Cheers,
> Rodrigo
>
>
> On 3/17/10 11:27 PM, "chaitanya krishna" <chaitanyavv.iiith@gmail.com>
> wrote:
>
> > In case you want to use hadoop-core.jar containing your modifications in
> > common, i think the following works:
> >
> > 1. ant mvn-install in common project. This will create hadoop-core.jar
> and
> > installs it in the local m2 repository.
> > 2. ant test -Dresolvers=internal in mapreduce project. This will make
> sure
> > that the hadoop-core.jar is taken from local m2 repository.
> >
> > Hope this helps.
> >
> > -Chaitanya.
> >
> > On Thu, Mar 18, 2010 at 11:53 AM, Rodrigo Schmidt <rodrigo@facebook.com
> >wrote:
> >
> >>
> >> I tried that but my logs still didn't show up. To test if it really
> worked,
> >> I disabled my Internet connection and tried it, which failed. Did I miss
> >> anything?
> >>
> >>
> >> Output below:
> >>
> >>
> >> $ant -Doffline=yes test
> >> Buildfile: build.xml
> >>
> >> ivy-download:
> >>
> >> ivy-init-dirs:
> >>
> >> ivy-probe-antlib:
> >>
> >> ivy-init-antlib:
> >>
> >> ivy-init:
> >> [ivy:configure] :: Ivy 2.0.0-rc2 - 20081028224207 ::
> >> http://ant.apache.org/ivy/ ::
> >> :: loading settings :: file =
> >> /Users/rschmidt/workspace/hadoop-mapreduce-trunk/ivy/ivysettings.xml
> >>
> >> ivy-resolve-common:
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve] You probably access the destination server through a proxy
> >> server that is not well configured.
> >> [ivy:resolve]
> >> [ivy:resolve] :: problems summary ::
> >> [ivy:resolve] :::: WARNINGS
> >> [ivy:resolve]     Host repository.apache.org not found.
> >> url=
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>
> hadoop/hadoop-core/0.22.0-SNAPSHOT/maven-metadata.xml<https://repository.apac
> >>
> he.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoop-core/0.22.0
> >> -SNAPSHOT/maven-metadata.xml>
> >> [ivy:resolve]     Host repository.apache.org not found.
> >> url=
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >> hadoop/hadoop-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.pom<
> https://re
> >>
> pository.apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoo
> >> p-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]     Host repository.apache.org not found.
> >> url=
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >> hadoop/hadoop-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.jar<
> https://re
> >>
> pository.apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoo
> >> p-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]     Host repo1.maven.org not found.
> >> url=
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/0.22.0-SNAPS
> >> HOT/hadoop-core-0.22.0-SNAPSHOT.pom<
> http://repo1.maven.org/maven2/org/apache/
> >> hadoop/hadoop-core/0.22.0-SNAPS%0AHOT/hadoop-core-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]     Host repo1.maven.org not found.
> >> url=
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/0.22.0-SNAPS
> >> HOT/hadoop-core-0.22.0-SNAPSHOT.jar<
> http://repo1.maven.org/maven2/org/apache/
> >> hadoop/hadoop-core/0.22.0-SNAPS%0AHOT/hadoop-core-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]         module not found:
> >> org.apache.hadoop#hadoop-core;0.22.0-SNAPSHOT
> >> [ivy:resolve]     ==== apache-snapshot: tried
> >> [ivy:resolve]
> >>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >> op/hadoop-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.pom<
> https://reposi
> >>
> tory.apache.org/content/repositories/snapshots/org/apache/hado%0Aop/hadoop-co
> >> re/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]       -- artifact
> >> org.apache.hadoop#hadoop-core;0.22.0-SNAPSHOT!hadoop-core.jar:
> >> [ivy:resolve]
> >>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >> op/hadoop-core/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.jar<
> https://reposi
> >>
> tory.apache.org/content/repositories/snapshots/org/apache/hado%0Aop/hadoop-co
> >> re/0.22.0-SNAPSHOT/hadoop-core-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]     ==== maven2: tried
> >> [ivy:resolve]
> >>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/0.22.0-SNAPSHOT/
> >> hadoop-core-0.22.0-SNAPSHOT.pom<
> http://repo1.maven.org/maven2/org/apache/hado
> >> op/hadoop-core/0.22.0-SNAPSHOT/%0Ahadoop-core-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]       -- artifact
> >> org.apache.hadoop#hadoop-core;0.22.0-SNAPSHOT!hadoop-core.jar:
> >> [ivy:resolve]
> >>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/0.22.0-SNAPSHOT/
> >> hadoop-core-0.22.0-SNAPSHOT.jar<
> http://repo1.maven.org/maven2/org/apache/hado
> >> op/hadoop-core/0.22.0-SNAPSHOT/%0Ahadoop-core-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]     Host repository.apache.org not found.
> >> url=
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >> hadoop/hadoop-core-test/0.22.0-SNAPSHOT/maven-metadata.xml<
> https://repository
> >> .
> apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoop-core-t
> >> est/0.22.0-SNAPSHOT/maven-metadata.xml>
> >> [ivy:resolve]     Host repository.apache.org not found.
> >> url=
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>
> >>
> hadoop/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom<
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/%0Aha
> >>
> doop/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]     Host repository.apache.org not found.
> >> url=
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>
> >>
> hadoop/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar<
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/%0Aha
> >>
> doop/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]     Host repo1.maven.org not found.
> >> url=
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core-test/0.22.0-
> >> SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom<
> http://repo1.maven.org/maven2/o
> >>
> rg/apache/hadoop/hadoop-core-test/0.22.0-%0ASNAPSHOT/hadoop-core-test-0.22.0-
> >> SNAPSHOT.pom>
> >> [ivy:resolve]     Host repo1.maven.org not found.
> >> url=
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core-test/0.22.0-
> >> SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar<
> http://repo1.maven.org/maven2/o
> >>
> rg/apache/hadoop/hadoop-core-test/0.22.0-%0ASNAPSHOT/hadoop-core-test-0.22.0-
> >> SNAPSHOT.jar>
> >> [ivy:resolve]         module not found:
> >> org.apache.hadoop#hadoop-core-test;0.22.0-SNAPSHOT
> >> [ivy:resolve]     ==== apache-snapshot: tried
> >> [ivy:resolve]
> >>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >>
> op/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom<http
> >> s://
> repository.apache.org/content/repositories/snapshots/org/apache/hado%0Aop
> >> /hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]       -- artifact
> >> org.apache.hadoop#hadoop-core-test;0.22.0-SNAPSHOT!hadoop-core-test.jar:
> >> [ivy:resolve]
> >>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >>
> op/hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar<http
> >> s://
> repository.apache.org/content/repositories/snapshots/org/apache/hado%0Aop
> >> /hadoop-core-test/0.22.0-SNAPSHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]     ==== maven2: tried
> >> [ivy:resolve]
> >>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core-test/0.22.0-SNAP
> >> SHOT/hadoop-core-test-0.22.0-SNAPSHOT.pom<
> http://repo1.maven.org/maven2/org/a
> >>
> pache/hadoop/hadoop-core-test/0.22.0-SNAP%0ASHOT/hadoop-core-test-0.22.0-SNAP
> >> SHOT.pom>
> >> [ivy:resolve]       -- artifact
> >> org.apache.hadoop#hadoop-core-test;0.22.0-SNAPSHOT!hadoop-core-test.jar:
> >> [ivy:resolve]
> >>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core-test/0.22.0-SNAP
> >> SHOT/hadoop-core-test-0.22.0-SNAPSHOT.jar<
> http://repo1.maven.org/maven2/org/a
> >>
> pache/hadoop/hadoop-core-test/0.22.0-SNAP%0ASHOT/hadoop-core-test-0.22.0-SNAP
> >> SHOT.jar>
> >> [ivy:resolve]     Host repository.apache.org not found.
> >> url=
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >>
> hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/maven-metadata.xml<https://repository.apac
> >>
> he.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoop-hdfs/0.22.0
> >> -SNAPSHOT/maven-metadata.xml>
> >> [ivy:resolve]     Host repository.apache.org not found.
> >> url=
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >> hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom<
> https://re
> >>
> pository.apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoo
> >> p-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]     Host repository.apache.org not found.
> >> url=
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/
> >> hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar<
> https://re
> >>
> pository.apache.org/content/repositories/snapshots/org/apache/%0Ahadoop/hadoo
> >> p-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]     Host repo1.maven.org not found.
> >> url=
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPS
> >> HOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom<
> http://repo1.maven.org/maven2/org/apache/
> >> hadoop/hadoop-hdfs/0.22.0-SNAPS%0AHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]     Host repo1.maven.org not found.
> >> url=
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPS
> >> HOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar<
> http://repo1.maven.org/maven2/org/apache/
> >> hadoop/hadoop-hdfs/0.22.0-SNAPS%0AHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]         module not found:
> >> org.apache.hadoop#hadoop-hdfs;0.22.0-SNAPSHOT
> >> [ivy:resolve]     ==== apache-snapshot: tried
> >> [ivy:resolve]
> >>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >> op/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom<
> https://reposi
> >>
> tory.apache.org/content/repositories/snapshots/org/apache/hado%0Aop/hadoop-hd
> >> fs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]       -- artifact
> >> org.apache.hadoop#hadoop-hdfs;0.22.0-SNAPSHOT!hadoop-hdfs.jar:
> >> [ivy:resolve]
> >>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/hado
> >> op/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar<
> https://reposi
> >>
> tory.apache.org/content/repositories/snapshots/org/apache/hado%0Aop/hadoop-hd
> >> fs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]     ==== maven2: tried
> >> [ivy:resolve]
> >>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/
> >> hadoop-hdfs-0.22.0-SNAPSHOT.pom<
> http://repo1.maven.org/maven2/org/apache/hado
> >> op/hadoop-hdfs/0.22.0-SNAPSHOT/%0Ahadoop-hdfs-0.22.0-SNAPSHOT.pom>
> >> [ivy:resolve]       -- artifact
> >> org.apache.hadoop#hadoop-hdfs;0.22.0-SNAPSHOT!hadoop-hdfs.jar:
> >> [ivy:resolve]
> >>
> >>
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/
> >> hadoop-hdfs-0.22.0-SNAPSHOT.jar<
> http://repo1.maven.org/maven2/org/apache/hado
> >> op/hadoop-hdfs/0.22.0-SNAPSHOT/%0Ahadoop-hdfs-0.22.0-SNAPSHOT.jar>
> >> [ivy:resolve]         ::::::::::::::::::::::::::::::::::::::::::::::
> >> [ivy:resolve]         ::          UNRESOLVED DEPENDENCIES         ::
> >> [ivy:resolve]         ::::::::::::::::::::::::::::::::::::::::::::::
> >> [ivy:resolve]         :: org.apache.hadoop#hadoop-core;0.22.0-SNAPSHOT:
> not
> >> found
> >> [ivy:resolve]         ::
> >> org.apache.hadoop#hadoop-core-test;0.22.0-SNAPSHOT:
> >> not found
> >> [ivy:resolve]         :: org.apache.hadoop#hadoop-hdfs;0.22.0-SNAPSHOT:
> not
> >> found
> >> [ivy:resolve]         ::::::::::::::::::::::::::::::::::::::::::::::
> >> [ivy:resolve]
> >> [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> >>
> >>
> >>
> >> On 3/17/10 11:14 PM, "Amar Kamat" <amarrk@yahoo-inc.com> wrote:
> >>
> >>> Use -Doffline=yes switch while running the tests.
> >>> Amar
> >>>
> >>> On 3/18/10 11:32 AM, "Rodrigo Schmidt" <rodrigo@facebook.com> wrote:
> >>>
> >>>
> >>>
> >>> How do I disable the ivy download to run unit tests? I need to replace
> >>> hadoop-core.jar for one I created with some extra logging to debug a
> unit
> >> test
> >>> for mapreduce.
> >>>
> >>> Right now I'm just erasing some lines from build.xml, but I'm quite
> sure
> >> there
> >>> is an easier way to do that.
> >>>
> >>> Thanks,
> >>> Rodrigo
> >>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message