From dev-return-32627-archive-asf-public=cust-asf.ponee.io@geode.apache.org Mon Dec 2 17:33:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2161618064E for ; Mon, 2 Dec 2019 18:33:02 +0100 (CET) Received: (qmail 82366 invoked by uid 500); 2 Dec 2019 17:33:01 -0000 Mailing-List: contact dev-help@geode.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@geode.apache.org Delivered-To: mailing list dev@geode.apache.org Received: (qmail 82350 invoked by uid 99); 2 Dec 2019 17:33:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Dec 2019 17:33:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 88D32C055B for ; Mon, 2 Dec 2019 17:33:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.489 X-Spam-Level: X-Spam-Status: No, score=-0.489 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=0.2, KAM_DMARC_STATUS=0.01, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id qjWem_MzKcuF for ; Mon, 2 Dec 2019 17:32:58 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=148.163.153.148; helo=mx0b-00296801.pphosted.com; envelope-from=jhuynh@pivotal.io; receiver= Received: from mx0b-00296801.pphosted.com (mx0b-00296801.pphosted.com [148.163.153.148]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 13F56BC52D for ; Mon, 2 Dec 2019 17:32:57 +0000 (UTC) Received: from pps.filterd (m0114584.ppops.net [127.0.0.1]) by mx0b-00296801.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id xB2HUk0X026006 for ; Mon, 2 Dec 2019 17:32:57 GMT Received: from mail-vk1-f198.google.com (mail-vk1-f198.google.com [209.85.221.198]) by mx0b-00296801.pphosted.com with ESMTP id 2wkg6k9dst-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 02 Dec 2019 17:32:57 +0000 Received: by mail-vk1-f198.google.com with SMTP id y189so193039vka.14 for ; Mon, 02 Dec 2019 09:32:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=zmJ9WA2077A9rZjkyqLTFRQ+8unLHpn84Vkm9p/1oJ0=; b=dIL8U3o/OudVd0kyQQ1iuSk4bN/XHm0D0jOOFuzk8emFZyX7IfjjXY7hbg2NKu+ic4 BxFn9bKtoBEcXNwF9Y9swuGFavSM5pb39U4xtnYjU4+FeFzxOBE14zkYH0bt/8wJ0Vrq TrrUbLpkqyOSumRVua2js08LKJndFO8LqL/eVzCkfVZIjWT2VDfUiYl0OwEQSnulev4S JvesdqI39b5yLe6Ba+pUU89Ad1YIX6kfXVxoAF94iJjPNiZfHdw4kn0GWH3pjfoQ2Djr 6SKiP4PTyOF9zpKpoXlOUX+JbZOQnKBqvnw6k8edC0TBGCWSid6DlN66xrXUUEzzuG0A LeHQ== X-Gm-Message-State: APjAAAUfPElYQ3j59Y4G4PN2K2+BhbXo6u2/Xn0fkWW7aMwAeWkU6rOy o+2QXUGawnD5DTkj62Qx1QBSlDkacz0tntsURkLJW7++wwByXc4UoxMq1kRXppetRc3RgpYdUvE l8YPRuKbkSsLLfo6VFuR2vDMjAD+KOLpKQ3gwT0xs1NICAfyznIvmsD0= X-Received: by 2002:a67:f78b:: with SMTP id j11mr56487vso.104.1575307976285; Mon, 02 Dec 2019 09:32:56 -0800 (PST) X-Google-Smtp-Source: APXvYqx2hNqLS+ih/6D1jJaLLog+T1WhvpYK10TPX/QAFWbEgdCpui59bgsnxV8C00VdEd02j83sD+iE0amY3LQ7S2E= X-Received: by 2002:a67:f78b:: with SMTP id j11mr56465vso.104.1575307975958; Mon, 02 Dec 2019 09:32:55 -0800 (PST) MIME-Version: 1.0 References: <78464E50-4F13-4AAC-BBCF-D95019EDAB63@pivotal.io> <5C0F7D22-64D5-4732-AE0B-610B1DEB388C@pivotal.io> <65290214-3416-4992-A217-715C718EE109@pivotal.io> In-Reply-To: From: Jason Huynh Date: Mon, 2 Dec 2019 09:32:44 -0800 Message-ID: Subject: Re: Odg: Lucene upgrade To: geode Content-Type: multipart/alternative; boundary="00000000000051bcfd0598bbf8c6" X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,18.0.572 definitions=2019-12-02_04:2019-11-29,2019-12-02 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 spamscore=0 priorityscore=1501 suspectscore=0 phishscore=0 adultscore=0 clxscore=1015 mlxlogscore=999 impostorscore=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1910280000 definitions=main-1912020147 --00000000000051bcfd0598bbf8c6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Mario, Sorry I reread the original email and see that the exception points to a different problem.. I think your fix addresses an old version seeing an unknown new lucene format, which looks good. The following exception looks like it's the new lucene library not being able to read the older files (Just a guess from the message)... Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(segments_1)): 6 (needs to be between 7 and 9). This version of Lucene only supports indexes created with release 6.0 and later. The upgrade is from 6.6.2 -> 8.x though, so I am not sure if the message is incorrect (stating needs to be release 6.0 and later) or if it requires an intermediate upgrade between 6.6.2 -> 7.x -> 8. On Mon, Dec 2, 2019 at 2:00 AM Mario Kevo wrote: > > I started with implementation of Option-1. > As I understood the idea is to block all puts(put them in the queue) unti= l > all members are upgraded. After that it will process all queued events. > > I tried with Dan's proposal to check on start of > LuceneEventListener.process() if all members are upgraded, also changed > test to verify lucene indexes only after all members are upgraded, but go= t > the same error with incompatibilities between lucene versions. > Changes are visible on https://github.com/apache/geode/pull/4198. > > Please add comments and suggestions. > > BR, > Mario > > > ________________________________ > =C5=A0alje: Xiaojian Zhou > Poslano: 7. studenog 2019. 18:27 > Prima: geode > Predmet: Re: Lucene upgrade > > Oh, I misunderstood option-1 and option-2. What I vote is Jason's option-= 1. > > On Thu, Nov 7, 2019 at 9:19 AM Jason Huynh wrote: > > > Gester, I don't think we need to write in the old format, we just need > the > > new format not to be written while old members can potentially read the > > lucene files. Option 1 can be very similar to Dan's snippet of code. > > > > I think Option 2 is going to leave a lot of people unhappy when they ge= t > > stuck with what Mario is experiencing right now and all we can say is > "you > > should have read the doc". Not to say Option 2 isn't valid and it's > > definitely the least amount of work to do, I still vote option 1. > > > > On Wed, Nov 6, 2019 at 5:16 PM Xiaojian Zhou wrote: > > > > > Usually re-creating region and index are expensive and customers are > > > reluctant to do it, according to my memory. > > > > > > We do have an offline reindex scripts or steps (written by Barry?). I= f > > that > > > could be an option, they can try that offline tool. > > > > > > I saw from Mario's email, he said: "I didn't found a way to write > lucene > > in > > > older format. They only support > > > reading old format indexes with newer version by using lucene-backwar= d- > > > codec." > > > > > > That's why I think option-1 is not feasible. > > > > > > Option-2 will cause the queue to be filled. But usually customer will > > hold > > > on, silence or reduce their business throughput when > > > doing rolling upgrade. I wonder if it's a reasonable assumption. > > > > > > Overall, after compared all the 3 options, I still think option-2 is > the > > > best bet. > > > > > > Regards > > > Gester > > > > > > > > > On Wed, Nov 6, 2019 at 3:38 PM Jacob Barrett > > wrote: > > > > > > > > > > > > > > > > On Nov 6, 2019, at 3:36 PM, Jason Huynh wrote= : > > > > > > > > > > Jake - there is a side effect to this in that the user would have > to > > > > > reimport all their data into the user defined region too. Client > > apps > > > > > would also have to know which of the regions to put into.. also, = I > > may > > > be > > > > > misunderstanding this suggestion, completely. In either case, I'= ll > > > > support > > > > > whoever implements the changes :-P > > > > > > > > Ah=E2=80=A6 there isn=E2=80=99t a way to re-index the existing data= . Eh=E2=80=A6 just a > > thought. > > > > > > > > -Jake > > > > > > > > > > > > > > --00000000000051bcfd0598bbf8c6--