Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 54068 invoked from network); 9 Sep 2008 04:37:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 9 Sep 2008 04:37:18 -0000 Received: (qmail 26864 invoked by uid 500); 9 Sep 2008 04:37:11 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 26828 invoked by uid 500); 9 Sep 2008 04:37:11 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 26817 invoked by uid 99); 9 Sep 2008 04:37:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Sep 2008 21:37:11 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lecompte@gmail.com designates 209.85.198.225 as permitted sender) Received: from [209.85.198.225] (HELO rv-out-0506.google.com) (209.85.198.225) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Sep 2008 04:36:13 +0000 Received: by rv-out-0506.google.com with SMTP id k40so1566006rvb.29 for ; Mon, 08 Sep 2008 21:36:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=VEyvj3kLOB5sW2B/VizyXmNPEKmwMpsEAEI9N+HIMa0=; b=sjVyGLylZkdfhseqDxJaL8PrFsyKOEpsHnaI9udOUmZYA1eBahxSM1NKvF2y4cLjY9 D4QsowW+MmZl99uYDEnDskFN693Sxgh6VEI1y0PB2IErX8+0CVZ6TlG+kVfSjM9sLRhk MlYFmrSzwyEb290X/A8dk2bR4l4TOWws/dY64= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=MPuF6fpc0Ul0Tri5d66fSbq6LwpdyQ0uRKxa8phTQJzLRfrfQbsMy/chdf8dfCdTpI 3vX5vQsSKhutJMuXUl7ZvFQ/s+Sd7jhKbqQdViM9VbYH5npR2+QHdZhnhXDfbSwGQG0E 3WcQPOTlMJJ4MBzTew9pEWpjdX9B8uYKWPFb8= Received: by 10.141.43.5 with SMTP id v5mr9387989rvj.216.1220935005054; Mon, 08 Sep 2008 21:36:45 -0700 (PDT) Received: by 10.141.98.6 with HTTP; Mon, 8 Sep 2008 21:36:45 -0700 (PDT) Message-ID: <6acdaf2a0809082136r2a3d565fp3029f7b51d9b880b@mail.gmail.com> Date: Tue, 9 Sep 2008 00:36:45 -0400 From: "Ryan LeCompte" To: core-user@hadoop.apache.org Subject: Issue in reduce phase with SortedMapWritable and custom Writables as values MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Virus-Checked: Checked by ClamAV on apache.org Hello, I'm attempting to use a SortedMapWritable with a LongWritable as the key and a custom implementation of org.apache.hadoop.io.Writable as the value. I notice that my program works fine when I use another primitive wrapper (e.g. Text) as the value, but fails with the following exception when I use my custom Writable instance: 2008-09-08 23:25:02,072 INFO org.apache.hadoop.mapred.ReduceTask: Initiating in-memory merge with 1 segments... 2008-09-08 23:25:02,077 INFO org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2008-09-08 23:25:02,077 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 5492 bytes 2008-09-08 23:25:02,099 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200809082247_0005_r_000000_0 Merge of the inmemory files threw a n exception: java.io.IOException: Intermedate merge failed at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2133) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2064) Caused by: java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80) at org.apache.hadoop.io.SortedMapWritable.readFields(SortedMapWritable.java:179) ... I noticed that the AbstractMapWritable class has a protected "addToMap(Class clazz)" method. Do I somehow need to let my SortedMapWritable instance know about my custom Writable value? I've properly implemented the custom Writable object (it just contains a few primitives, like longs and ints). Any insight is appreciated. Thanks, Ryan