From dev-return-4127-archive-asf-public=cust-asf.ponee.io@hudi.apache.org Wed Jun 23 03:20:56 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 6CE7C180669 for ; Wed, 23 Jun 2021 05:20:56 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 1369D61973 for ; Wed, 23 Jun 2021 03:20:54 +0000 (UTC) Received: (qmail 93071 invoked by uid 500); 23 Jun 2021 03:20:53 -0000 Mailing-List: contact dev-help@hudi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hudi.apache.org Delivered-To: mailing list dev@hudi.apache.org Received: (qmail 93023 invoked by uid 99); 23 Jun 2021 03:20:53 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2021 03:20:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 1B849C03D8 for ; Wed, 23 Jun 2021 03:20:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id Z8OzFdv1KK92 for ; Wed, 23 Jun 2021 03:20:51 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::52c; helo=mail-pg1-x52c.google.com; envelope-from=email2aakash@gmail.com; receiver= Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 50D077FFC4 for ; Wed, 23 Jun 2021 03:20:51 +0000 (UTC) Received: by mail-pg1-x52c.google.com with SMTP id y14so552052pgs.12 for ; Tue, 22 Jun 2021 20:20:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=c8hTuMoCMT+1/HwlMDIHc6necQZhy2NLKoc1tCzIIQk=; b=fBqbK1O+Jv0oe1bszHQTc3nT1ekrUNvcCcGH6EfQ2S5FanAkRKY71vjgYaAYt31y/f k9Oi6mNvECax7weXF94aSjY9RqQPmsZw85+z7fyofsm3hqlzIO1yyY1aM74eNZZtMy1L dG5ssO39Ii6Lb0N/1BlkkuMzMhSp9KTrpXqoKh1ovSDnyHnR1ALQEZ8Qfcu/5H6muGq/ B+B4mFgDi/VK7dqMVM3ZkBsobnY82035Bc0MvfOd577MqKF/u8G33RFieM26vkVLGRZ2 tOYxRN6EVJXz7VRVF3CAvRnO6fPRXXT1GbknHpwmRkBB9GxQm2jVNTwk/tPsKgFjMXRV PfqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=c8hTuMoCMT+1/HwlMDIHc6necQZhy2NLKoc1tCzIIQk=; b=lXKCGbHC8DJVphrl42/+N+5+WRCazpMK04ZLUHiQVobzixNtA0z6WFRj21mEalFoJT 9faewjfQrqC6dBHIgUEV2bwPN4ZuUUTZIyUHqTFlETq+H1bzk6xL4QpdqUat1UrW+yM1 0dgbUDIUVJNlY8NYMyj1rJBvbWBkWfWM78h7ypSZmmD0g3VgonmHnSTDu1HuQf1Tza8C K68qRG2uSOMi0foBNZSCd/yKRtklcQo9HMO7zxURw5AJr5+pi1ndtiNkI9lVSaC+eH1n Bdmz+c6HXLikfKdHgdNKFG+tu0DKDMKxaIHtC/bIwTJvpRORD4Pq8v5CfH6LBY8hgN5f KdSA== X-Gm-Message-State: AOAM531tqPst6ZHQVOsOj6+TsCAM/6I9qdniW2K06bwnGVI/PUNEacVM l9v67LyIqjPpSq+o4p+gxoD9QDPAlvCtnIoIpt89Z2ICZuzPiQ== X-Google-Smtp-Source: ABdhPJz79h8RxXtBM4oDTZNre1/LEt/vk8fnaFdZJgHvxC7rXziT6+tN70HuHuSP9Yy3peL/LKG3ZxUbofiZ6lR3XvI= X-Received: by 2002:aa7:8749:0:b029:2f1:3dd0:674 with SMTP id g9-20020aa787490000b02902f13dd00674mr6744370pfo.65.1624418443421; Tue, 22 Jun 2021 20:20:43 -0700 (PDT) MIME-Version: 1.0 From: aakash aakash Date: Tue, 22 Jun 2021 20:20:32 -0700 Message-ID: Subject: issue while reading archived commit written by 0.5 version with 0.8 version To: dev@hudi.apache.org Content-Type: multipart/alternative; boundary="00000000000049695705c5666473" --00000000000049695705c5666473 Content-Type: text/plain; charset="UTF-8" Hi, I am trying to use Hudi 0.8 with Spark 3.0 in my prod environment and earlier we were running Hudi 0.5 with Spark 2.4.4. While updating a very old index, I am getting this error : *from the logs it seem its error out while reading this file : hudi/.hoodie/archived/.commits_.archive.119_1-0-1 in s3* 21/06/22 19:18:06 ERROR HoodieTimelineArchiveLog: Failed to archive commits, .commit file: 20200715192915.rollback.inflight java.io.IOException: Not an Avro data file at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:50) at org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata(TimelineMetadataUtils.java:175) at org.apache.hudi.client.utils.MetadataConversionUtils.createMetaWrapper(MetadataConversionUtils.java:84) at org.apache.hudi.table.HoodieTimelineArchiveLog.convertToAvroRecord(HoodieTimelineArchiveLog.java:370) at org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:311) at org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:128) at org.apache.hudi.client.AbstractHoodieWriteClient.postCommit(AbstractHoodieWriteClient.java:430) at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:186) at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:121) at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:479) Is this a backward compatibility issue? I have deleted a few archive files but the problem is persisting so it does not look like a file corruption issue. Regards, Aakash --00000000000049695705c5666473--