Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4EA85D07E for ; Sat, 4 Aug 2012 15:45:23 +0000 (UTC) Received: (qmail 10109 invoked by uid 500); 4 Aug 2012 15:45:19 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 10047 invoked by uid 500); 4 Aug 2012 15:45:19 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 10039 invoked by uid 99); 4 Aug 2012 15:45:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Aug 2012 15:45:19 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [212.227.17.9] (HELO moutng.kundenserver.de) (212.227.17.9) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Aug 2012 15:45:10 +0000 Received: from mafr.de (krlh-5f720d93.pool.mediaWays.net [95.114.13.147]) by mrelayeu.kundenserver.de (node=mrbap3) with ESMTP (Nemesis) id 0M2dAz-1TnB9o3KdR-00sPJw; Sat, 04 Aug 2012 17:44:50 +0200 Date: Sat, 4 Aug 2012 17:44:47 +0200 From: Matthias Friedrich To: common-user@hadoop.apache.org Subject: Maven dependency problems Message-ID: <20120804154435.GC11145@mafr.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Provags-ID: V02:K0:W/fv3QklxR5Wm3WTaRHVxeEcC8p1k11hJMtyePXtkGw eXBM71/Rc+C6GbhYQE8DFTh6Bw/sjcT1eU0saw7jpScTj2W3Lc 0xSscDr2CBdQc55Lpvjqx8INAv0nBP3mcmE3HBvUup2VBOguDp Np4UxItkTLys08qlFlvXVB97hHbXamSfva/qWy49+zeGKU8M1X za1xJUhlkK9ltiGvb3fw0aaoahCcsXvD3v7+GEB6drQsdW0X5Y hD6X9TsyCKKwner2PMvBh0QTYVDOAwnc50/nAdQYDEcjik284U rl7B/xkHIfen2xMCisg6m3iSEJ+4oqAuIIw6udgm04KQQeSvR2 MZKIBj84PwfLrsYqmO28= Hi, I'm currently trying to fix Maven dependencies for Crunch and ran into trouble with the POM for hadoop-core 1.0.3. It looks like the Maven dependencies are different from the actual dependencies at runtime. As a result, bugs caused by dependency conflicts won't show up until runtime, makeing unit and integration tests in Maven useless for detecting this class of problems. Unfortunately, these bugs tend to be quite nasty. For example, Hadoop ships a very old version of slf4j. If my job includes a newer one this can lead to MethodNotFoundExceptions. Since the dependency to slf4j is undeclared in hadoop-core, there is no way to detect this using Maven dependency reports. Other conflicts may lead to different behavior which is a lot harder to diagnose. What's the best way to get this fixed? Or is there something I'm missing? Below I attached a report of the differences I found. I didn't list the correct ones. Thanks, Matthias On the classpath as per org.apache.hadoop:hadoop-core:jar:1.0.3 --------------------------------------------------------------- ant-1.6.5.jar commons-logging-1.0.3.jar jackson-core-asl-1.0.1.jar jackson-mapper-asl-1.0.1.jar jets3t-0.7.1.jar jsp-2.1-6.1.14.jar jsp-api-2.1-6.1.14.jar junit-3.8.1.jar kfs-0.3.jar servlet-api-2.5-6.1.14.jar Actually on the classpath as per lib directory of hadoop-1.0.3-bin.tar.gz ------------------------------------------------------------------------- asm-3.2.jar aspectjrt-1.6.5.jar aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar commons-io-2.1.jar commons-logging-1.1.1.jar commons-logging-api-1.0.4.jar hadoop-capacity-scheduler-1.0.3.jar hadoop-fairscheduler-1.0.3.jar hadoop-thriftfs-1.0.3.jar jackson-core-asl-1.8.8.jar jackson-mapper-asl-1.8.8.jar jdeb-0.8.jar jersey-core-1.8.jar jersey-json-1.8.jar jersey-server-1.8.jar jets3t-0.6.1.jar jsch-0.1.42.jar junit-4.5.jar kfs-0.2.2.jar log4j-1.2.15.jar mockito-all-1.8.5.jar slf4j-api-1.4.3.jar slf4j-log4j12-1.4.3.jar