Return-Path: X-Original-To: apmail-avro-user-archive@www.apache.org Delivered-To: apmail-avro-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2BC739C25 for ; Tue, 3 Apr 2012 08:29:10 +0000 (UTC) Received: (qmail 77135 invoked by uid 500); 3 Apr 2012 08:29:10 -0000 Delivered-To: apmail-avro-user-archive@avro.apache.org Received: (qmail 76784 invoked by uid 500); 3 Apr 2012 08:29:07 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 76746 invoked by uid 99); 3 Apr 2012 08:29:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Apr 2012 08:29:06 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of markus.resch@teamaol.com designates 205.188.105.146 as permitted sender) Received: from [205.188.105.146] (HELO imr-da04.mx.aol.com) (205.188.105.146) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Apr 2012 08:28:58 +0000 Received: from aoldtcmei32.ad.aol.aoltw.net (aoldtcmei32.office.aol.com [10.180.121.111]) by imr-da04.mx.aol.com (8.14.1/8.14.1) with ESMTP id q338SVXE024492; Tue, 3 Apr 2012 04:28:31 -0400 Received: from AOLFRRMEC32.ad.aol.aoltw.net (10.149.226.44) by aoldtcmei32.ad.aol.aoltw.net (10.180.121.111) with Microsoft SMTP Server (TLS) id 14.2.283.3; Tue, 3 Apr 2012 04:28:31 -0400 Received: from [10.168.204.55] (172.17.52.163) by aolfrrmec32.ad.aol.aoltw.net (10.149.226.44) with Microsoft SMTP Server id 14.2.283.3; Tue, 3 Apr 2012 09:28:29 +0100 Subject: Sync Marker Issue while reading AVRO files writen with FLUME with PIG From: Markus Resch To: , Content-Type: text/plain; charset="UTF-8" Organization: AdTech Date: Tue, 3 Apr 2012 10:28:29 +0200 Message-ID: <1333441709.3055.40.camel@mresch.office.aol.com> MIME-Version: 1.0 X-Mailer: Evolution 2.30.3 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.52.163] Hey everyone, we're facing a problem while reading AVRO files written with FLUME using the AVRO Java API 1.5.4 into a HADOOP cluster. The Avro Data Store complains about missing sync marker. Investigating the problem shows us, that's perfectly right. The sync marker is missing. Thus we have a block of the double size. Our software packets: rpm -qa | grep hadoop hadoop-0.20-namenode-0.20.2+923.142-1 hadoop-0.20-0.20.2+923.142-1 hadoop-0.20-native-0.20.2+923.142-1 hadoop-hive-0.7.1+42.27-2 hadoop-pig-0.8.1+28.18-1 This is pretty much all a basic cloudera CDH3 Update 2 Packaging installation with a patched PIG version which is CDH3 Update 3. Did anyone had a similar issue? Does this ring a bell? Thanks Markus