From user-return-4326-archive-asf-public=cust-asf.ponee.io@kylin.apache.org Thu May 16 08:38:28 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 924C018066B for ; Thu, 16 May 2019 10:38:27 +0200 (CEST) Received: (qmail 23259 invoked by uid 500); 16 May 2019 08:38:26 -0000 Mailing-List: contact user-help@kylin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kylin.apache.org Delivered-To: mailing list user@kylin.apache.org Received: (qmail 23249 invoked by uid 99); 16 May 2019 08:38:26 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 May 2019 08:38:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 393681810F1 for ; Thu, 16 May 2019 08:38:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.906 X-Spam-Level: * X-Spam-Status: No, score=1.906 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_EXCESS_BASE64=0.105, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=telecoming.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 7MCACIjlzDb7 for ; Thu, 16 May 2019 08:38:23 +0000 (UTC) Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B7B8F612AB for ; Thu, 16 May 2019 08:38:23 +0000 (UTC) Received: by mail-io1-f41.google.com with SMTP id g16so1906293iom.9 for ; Thu, 16 May 2019 01:38:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telecoming.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=YveLobeisM8wzZsshQDHjgeyYQ3CYckqqxjGWxb0hHI=; b=LkYd78TZYrvPiZkAOQYMiEDx2OMSHFTjJLcfYVeNTmTZuX0A1uJ0JgY19rcFAAdIqz mYi1VpPqvwNDFEoDlXeIkpzbMLCqGGc54TEu5V/xzjGrLw3o7gs7s4FUUC9KyAbJ+XT0 78SeYak0brm+zEktZH1rriyZ9K3eWGlSKm8Rc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=YveLobeisM8wzZsshQDHjgeyYQ3CYckqqxjGWxb0hHI=; b=AO22nkbPbdSUWwauMin6WcqhxxHd7f+Zdkt6Ma7rdx7ZF5eNT+A2a5gpK2s6ygNghx dShN/fKVWZ1MUIhSCmC2B1D0jCs2qateWqNUwvnnxe7xLYJ5l1Jei9BiV1hUQtpTw/Ik YcWTPAuUj7SliSkKFd9OB+DFyzyJle+TP3yLESRMlK3efk8CsVVUm1ZpQIKryX1tT0Di GAHvIKlsf4QOz/UmmPRZmYx1i8AUH4JW07JwPQShpgEtGuzIwHTSfux6swZrIJS/7Xsm XzGXy8tx3jxUOjLU4SEjs70mrpx+/kMPg3//nCVzamRH9qiSJMkhME9X5PYIJzmMe9V3 Tm2w== X-Gm-Message-State: APjAAAU8XR5QUd6zv3cJamEVraVEwU0zm+JhmWel1sFzFNjatU5cn6bl FDMI/EkOYvPzEWggKxSJza/esUtm3xTB+Lu7m1vyAYJm X-Google-Smtp-Source: APXvYqzbGI4aRAsfqhiPSdFrfgKpw8wygdeEBRHG8J5/o++YkExupkrrXSo9poEmlu+pvtvRddL8KxJ7OXVGa137tPU= X-Received: by 2002:a5d:84ce:: with SMTP id z14mr29235760ior.107.1557995902774; Thu, 16 May 2019 01:38:22 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?ScOxaWdvIE1hcnTDrW5leg==?= Date: Thu, 16 May 2019 10:38:11 +0200 Message-ID: Subject: Re: Cleanup job not deleting hive intermediate table if created from view To: user@kylin.apache.org Content-Type: multipart/alternative; boundary="00000000000058c9ac0588fd3074" --00000000000058c9ac0588fd3074 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Na. Got it. The problem is this script is not locating beeline and so it cannot be launched. Since it's launched from cron, it does not heritage environment and so does not know where beeline is located. Fixed. I'm gonna close the Jira issue. El jue., 16 may. 2019 a las 10:17, I=C3=B1igo Mart=C3=ADnez () escribi=C3=B3: > Hi Na. > > Yes, I've read it. It's a bit strange. I will check it in order to find i= t > the problem is not the cleanup task itself but the script that launches i= t > or something else. > > El mi=C3=A9., 15 may. 2019 a las 15:20, Na Zhai () > escribi=C3=B3: > >> Hi, I=C3=B1igo Mart=C3=ADnez . >> >> >> >> I can not reproduce your situation. And I have answered in >> https://jira.apache.org/jira/browse/KYLIN-4002. >> >> >> >> =E5=8F=91=E9=80=81=E8=87=AA Windows 10 =E7=89=88=E9=82=AE=E4=BB=B6 =E5=BA=94=E7=94=A8 >> >> >> ------------------------------ >> *=E5=8F=91=E4=BB=B6=E4=BA=BA:* I=C3=B1igo Mart=C3=ADnez >> *=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4:* Tuesday, May 14, 2019 5:57:47 PM >> *=E6=94=B6=E4=BB=B6=E4=BA=BA:* user@kylin.apache.org >> *=E4=B8=BB=E9=A2=98:* Cleanup job not deleting hive intermediate table i= f created from >> view >> >> Just for your information, we have detected a problem that hurts >> performance in the long term due to a bug in a maintenance job. >> >> In our case, temporary intermediate tables in hive created from a hive >> view are not deleted by Kylin StorageCleanupJob. We realized about it >> because our backup process was stuck for hours when performing a >> differential copy from HDFS to S3. This was due to more than 1M addition= al >> files where present. >> >> I've opened a bug at Jira. >> https://jira.apache.org/jira/browse/KYLIN-4002 >> >> >> -- >> >> >> >> >> I=C3=B1igo Mart=C3=ADnez >> Systems Manager >> imartinez@telecoming.com >> >> >> >> >> >> >> [image: Telecoming - Make it digital] >> [image: 5000_empresas] >> [image: 1000_empresas] >> >> Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain | >> telecoming.com >> >> >> >> Este correo electr=C3=B3nico y sus archivos adjuntos est=C3=A1n dirigi= dos >> =C3=BAnicamente a la(s) direcci=C3=B3n(es) indicada(s) anteriormente. El= car=C3=A1cter >> confidencial, personal e intransferible del mismo est=C3=A1 protegido >> legalmente. Cualquier publicaci=C3=B3n, reproducci=C3=B3n, distribuci=C3= =B3n o >> retransmisi=C3=B3n no autorizada, ya sea completa o en parte, se encuent= ra >> prohibida. Si ha recibido este mensaje por equivocaci=C3=B3n, notif=C3= =ADquelo >> inmediatamente a la persona que lo ha enviado y borre el mensaje origina= l >> junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno. >> >> > > -- > > > > > I=C3=B1igo Mart=C3=ADnez > Systems Manager > imartinez@telecoming.com > > > > > > > [image: Telecoming - Make it digital] > [image: 5000_empresas] > [image: 1000_empresas] > > Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain | > telecoming.com > > > > Este correo electr=C3=B3nico y sus archivos adjuntos est=C3=A1n dirigid= os > =C3=BAnicamente a la(s) direcci=C3=B3n(es) indicada(s) anteriormente. El = car=C3=A1cter > confidencial, personal e intransferible del mismo est=C3=A1 protegido > legalmente. Cualquier publicaci=C3=B3n, reproducci=C3=B3n, distribuci=C3= =B3n o > retransmisi=C3=B3n no autorizada, ya sea completa o en parte, se encuentr= a > prohibida. Si ha recibido este mensaje por equivocaci=C3=B3n, notif=C3=AD= quelo > inmediatamente a la persona que lo ha enviado y borre el mensaje original > junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno. > > --=20 I=C3=B1igo Mart=C3=ADnez Systems Manager imartinez@telecoming.com [image: Telecoming - Make it digital] [image: 5000_empresas] [image: 1000_empresas] Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain | telecoming.com Este correo electr=C3=B3nico y sus archivos adjuntos est=C3=A1n dirigidos =C3=BAnicamente a la(s) direcci=C3=B3n(es) indicada(s) anteriormente. El ca= r=C3=A1cter confidencial, personal e intransferible del mismo est=C3=A1 protegido legalmente. Cualquier publicaci=C3=B3n, reproducci=C3=B3n, distribuci=C3=B3= n o retransmisi=C3=B3n no autorizada, ya sea completa o en parte, se encuentra prohibida. Si ha recibido este mensaje por equivocaci=C3=B3n, notif=C3=ADqu= elo inmediatamente a la persona que lo ha enviado y borre el mensaje original junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno. --00000000000058c9ac0588fd3074 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Na.

Got it. The problem is this scri= pt is not locating beeline and so it cannot be launched.
Since it= 's launched from cron, it does not heritage environment and so does not= know where beeline is located. Fixed.
I'm gonna close the Ji= ra issue.


El jue., 16 may. 2019 a las 10:17, I=C3=B1igo= Mart=C3=ADnez (<imartinez@t= elecoming.com>) escribi=C3=B3:
Hi Na.

Yes, I'= ;ve read it. It's a bit strange. I will check it in order to find it th= e problem is not the cleanup task itself but the script that launches it or= something else.

El mi=C3=A9., 15 may. 2019 a las 15:20, Na Zhai (<= na.zhai@kyligence= .io>) escribi=C3=B3:

Hi, I=C3=B1igo Mart=C3=ADnez.

=C2=A0

I can = not reproduce your situation. And I have answered in https://jira.apache.org/jira/browse/KYLIN-4002.<= /p>

=C2=A0

=E5=8F=91=E9=80=81=E8=87=AA Win= dows 10 =E7=89=88=E9=82=AE=E4=BB=B6=E5=BA=94=E7=94= =A8

=C2=A0

=E5=8F=91=E4=BB=B6=E4=BA=BA: I=C3=B1igo Mart=C3= =ADnez <im= artinez@telecoming.com>
=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: Tuesday, May 14, 2019 5:57:47 = PM
=E6=94=B6=E4=BB=B6=E4=BA=BA: user@kylin.apache.org
=E4=B8=BB=E9=A2=98: Cleanup job not deleting hive intermediate table= if created from view
=C2=A0
Just for your information, we have detected a problem that hurts perfo= rmance in the long term due to a bug in a maintenance job.

In our case, temporary intermediate tables in hive created from a hive= view are not deleted by Kylin StorageCleanupJob. We realized about it beca= use our backup process was stuck for hours when performing a differential c= opy from HDFS to S3. This was due to more than 1M additional files where present.=C2=A0

I've opened a bug at Jira.
https://jira.apache.org/jira/browse/KYLIN-4002


--
=C2=A0
=C2=A0
=C2=A0
I=C3=B1igo Mart=C3=ADnez
Systems Manager
imartinez@te= lecoming.com =C2=A0 =C2=A0
=C2=A0
=C2=A0
=C2=A0
=C2=A0
=C2=A0 3D"Telecoming =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
3D"5000_empresas"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
3D"1000_empresas"
=C2=A0
=C2=A0
Paseo de la Castellana, 95. Torr= e Europa, pl 16. 28046 Madrid, Spain | telecoming.com<= /font>
=C2=A0
=C2=A0
=C2=A0 Este correo electr=C3=B3nico y = sus archivos adjuntos est=C3=A1n dirigidos =C3=BAnicamente a la(s) direcci= =C3=B3n(es) indicada(s) anteriormente. El car=C3=A1cter confidencial, perso= nal e intransferible del mismo est=C3=A1 protegido legalmente. Cualquier publicaci=C3=B3n, reproducci=C3=B3n, distribuci=C3=B3n o retransmisi=C3=B3= n no autorizada, ya sea completa o en parte, se encuentra prohibida. Si ha = recibido este mensaje por equivocaci=C3=B3n, notif=C3=ADquelo inmediatament= e a la persona que lo ha enviado y borre el mensaje original junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.


--
=C2=A0
=C2=A0<= td>
=C2=A0
=C2=A0
Systems Manager
imartinez@telecoming.com=C2=A0=C2=A0
=C2=A0
=C2=A0
=
=C2=A0
=C2=A0
=C2=A03D"Telecoming=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =
3D"5000_empresas"
=C2=A0=C2=A0=C2=A0=C2=A0= =C2= =A0
=C2=A0
=C2=A0=
Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, = Spain |telecoming.com
=C2=A0
=C2=A0
=C2=A0Este correo electr=C3=B3nico y sus archi= vos adjuntos est=C3=A1n dirigidos =C3=BAnicamente a la(s) direcci=C3=B3n(es= ) indicada(s) anteriormente. El car=C3=A1cter confidencial, personal e intr= ansferible del mismo est=C3=A1 protegido legalmente. Cualquier publicaci=C3= =B3n, reproducci=C3=B3n, distribuci=C3=B3n o retransmisi=C3=B3n no autoriza= da, ya sea completa o en parte, se encuentra prohibida. Si ha recibido este= mensaje por equivocaci=C3=B3n, notif=C3=ADquelo inmediatamente a la person= a que lo ha enviado y borre el mensaje original junto con sus ficheros anex= os sin leerlo ni grabarlo en modo alguno.
<= /td>
<= /div>


--
=
=C2=A0<= tbody>
=C2=A0
=C2=A0<= td>
I=C3=B1igo Mart=C3=ADnez
Systems Manage= r
=
imartinez@telecoming.com=C2=A0=C2=A0=
=C2=A0
=C2=A0
=C2=A0
=C2=A0
= <= td width=3D"6" style=3D"font-size:12px;line-height:14px;white-space:nowrap"= >=C2=A0
=C2=A03D"Telecoming=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3D"5000_empresas"
=C2=A0=C2=A0=C2=A0=C2=A0
3D"=
=C2=A0
=C2=A0
Paseo de la Castell= ana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |telecoming.com
=C2=A0
<= tbody>
=C2= =A0
=C2=A0Este correo electr=C3=B3nico y sus archivos adjuntos est=C3=A1n dirigid= os =C3=BAnicamente a la(s) direcci=C3=B3n(es) indicada(s) anteriormente. El= car=C3=A1cter confidencial, personal e intransferible del mismo est=C3=A1 = protegido legalmente. Cualquier publicaci=C3=B3n, reproducci=C3=B3n, distri= buci=C3=B3n o retransmisi=C3=B3n no autorizada, ya sea completa o en parte,= se encuentra prohibida. Si ha recibido este mensaje por equivocaci=C3=B3n,= notif=C3=ADquelo inmediatamente a la persona que lo ha enviado y borre el = mensaje original junto con sus ficheros anexos sin leerlo ni grabarlo en mo= do alguno.
<= /td>
--00000000000058c9ac0588fd3074--