Return-Path: X-Original-To: apmail-incubator-crunch-dev-archive@minotaur.apache.org Delivered-To: apmail-incubator-crunch-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0EE5ADC63 for ; Mon, 9 Jul 2012 14:26:04 +0000 (UTC) Received: (qmail 85634 invoked by uid 500); 9 Jul 2012 14:26:04 -0000 Delivered-To: apmail-incubator-crunch-dev-archive@incubator.apache.org Received: (qmail 85585 invoked by uid 500); 9 Jul 2012 14:26:03 -0000 Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: crunch-dev@incubator.apache.org Delivered-To: mailing list crunch-dev@incubator.apache.org Received: (qmail 84942 invoked by uid 99); 9 Jul 2012 14:26:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Jul 2012 14:26:02 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of josh.wills@gmail.com designates 74.125.82.175 as permitted sender) Received: from [74.125.82.175] (HELO mail-we0-f175.google.com) (74.125.82.175) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Jul 2012 14:25:56 +0000 Received: by weyr6 with SMTP id r6so4443141wey.6 for ; Mon, 09 Jul 2012 07:25:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=MI2MPoAmKLyaAIzYqKYgYT1OOAlDPSxtLveNx0SNbHY=; b=rW9JiLKWcS05DBb+JhZcTzZj13V6s+eMYN80L65TxtuWxPwv65f8jpqAhXaoyvvwKh RFH2vhoCCQpV8mTZe1bs7XDhRPZaqc0c63lthyb9LRstGb9JrVjNrcsB7wMhTWyj+lvc qmXwQKDPZfsgo88djAODhktdAhF+l/gWIGdojnHuWjLTejoDYyAnv39w7R5UfUuUEbyC q4lwpQR2S+ycZA2CYzFb7Wj00iDSaodA40r/jqCXoHYn0uH3IwGez1X7/r5sbMwzDxit 8tI2dVyCQBgSnsLkUCUw5dqIx8lYfanQorZH9C+kUHPsQLOwjYYisv9d3MB1t/WhgljD NGag== Received: by 10.180.83.234 with SMTP id t10mr34484566wiy.0.1341843935375; Mon, 09 Jul 2012 07:25:35 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.201.159 with HTTP; Mon, 9 Jul 2012 07:25:15 -0700 (PDT) In-Reply-To: <4FFA9C7D.50804@xebia.com> References: <4FFA9C7D.50804@xebia.com> From: Josh Wills Date: Mon, 9 Jul 2012 07:25:15 -0700 Message-ID: Subject: Re: SeqFileReaderFactory give exception To: crunch-dev@incubator.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org SequenceFileTableSouce will let you read it the file as a PTable, which is probably the quickest way to get what you want. On Mon, Jul 9, 2012 at 1:55 AM, Rahul wrote: > Guys, > > I have a SequenceFile with LogWritable Keys and Text as values . I am using > SequenceFileSource with MRPipeline. But when I use MemPipeline it is giving > back the following exception. > > 3503 [main] INFO com.cloudera.crunch.io.seq.SeqFileReaderFactory - Error > reading from path: file:/home/rahul/software/crunch/sampleFile > java.io.IOException: wrong key class: org.apache.hadoop.io.ObjectWritable is > not class org.apache.hadoop.io.LongWritable > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1895) > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1947) > at > com.cloudera.crunch.io.seq.SeqFileReaderFactory$1.hasNext(SeqFileReaderFactory.java:68) > at > com.cloudera.crunch.io.CompositePathIterable$2.hasNext(CompositePathIterable.java:81) > > Now this is due to the fact that the file contains LongWritable Keys but it > is using a NullWritable to read them. This gives error in MemPipline only, > it works in the MRPipeline because the KeyClass is passed there using the > MapContext of Hadoop and thus it is the correct one. I modified the > SeqFileReaderFactory to pass the KeyClass also but is this the correct way > of doing so ? > > regards > Rahul