From issues-return-21056-archive-asf-public=cust-asf.ponee.io@kylin.apache.org Tue Oct 23 15:51:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9D0FE18067B for ; Tue, 23 Oct 2018 15:51:04 +0200 (CEST) Received: (qmail 28567 invoked by uid 500); 23 Oct 2018 13:51:03 -0000 Mailing-List: contact issues-help@kylin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kylin.apache.org Delivered-To: mailing list issues@kylin.apache.org Received: (qmail 28515 invoked by uid 99); 23 Oct 2018 13:51:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Oct 2018 13:51:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 226AA181BCD for ; Tue, 23 Oct 2018 13:51:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 5il3GjWqgqs5 for ; Tue, 23 Oct 2018 13:51:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id B4BDB5F4A9 for ; Tue, 23 Oct 2018 13:51:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id D1E52E1328 for ; Tue, 23 Oct 2018 13:51:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 59024252BA for ; Tue, 23 Oct 2018 13:51:00 +0000 (UTC) Date: Tue, 23 Oct 2018 13:51:00 +0000 (UTC) From: "Zhong Yanghong (JIRA)" To: issues@kylin.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Assigned] (KYLIN-2894) Change the query cache expiration strategy by signature checking MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/KYLIN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong reassigned KYLIN-2894: ------------------------------------- Assignee: Wang Ken (was: Zhong Yanghong) > Change the query cache expiration strategy by signature checking > ---------------------------------------------------------------- > > Key: KYLIN-2894 > URL: https://issues.apache.org/jira/browse/KYLIN-2894 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine > Reporter: Zhong Yanghong > Assignee: Wang Ken > Priority: Major > > Currently to invalid query cache, {{CacheService}} will either invoke {{cleanDataCache}} or {{cleanAllDataCache}}. Both methods will clear all of the query cache, which is very inefficient. In eBay PROD environment, there's around 400 cubing jobs per day, which means the query cache will be cleared very 4 minutes. Then we introduced a signature based cache invalidation strategy. The basic idea is as follows: > * Add a signature for {{SQLResponse}}, here we choose the cube last build time > * When fetch {{SQLResponse}} for cache, first check whether the signature is consistent. If not, this cached value is overdue and will be invalidate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)