Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
Message-Id: <201606101932.u5AJWWsM007040@ip-10-146-233-104.ec2.internal>
Date: Fri, 10 Jun 2016 19:32:32 +0000
From: "Huaisi Xu (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, dev@impala.incubator.apache.org
CC: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>,
        Dan Hecht <dhecht@cloudera.com>, Henry Robinson <henry@cloudera.com>,
        Bharath Vissapragada <bharathv@cloudera.com>
Reply-To: hxu@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-CR=5D=28cdh5-trunk=29_IMPALA-3499=3A_Split_catalog_update=0A?=
In-Reply-To: <gerrit.1463177606000.I176db25124a32944f2396ce8aafbed49cac95928@gerrit.cloudera.org>
References: <gerrit.1463177606000.I176db25124a32944f2396ce8aafbed49cac95928@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.2
archived-at: Fri, 10 Jun 2016 19:32:38 -0000

Huaisi Xu has posted comments on this change.

Change subject: IMPALA-3499: Split catalog update
......................................................................


Patch Set 17:

(7 comments)

No. I just used 10k as a limit. I do not have resources to test 2gb metadata.

http://gerrit.cloudera.org:8080/#/c/3067/17//COMMIT_MSG
Commit Message:

PS17, Line 10: 500MB
> The limit is 2GB, because it's a JVM limitation.
Done


http://gerrit.cloudera.org:8080/#/c/3067/17/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

PS17, Line 1252: current_topic_update_size_bytes
> batch_size_bytes is better, I think
Done


PS17, Line 1255: if (len > 100 * 1024 * 1024 /* 100MB */) {
               :         LOG(WARNING) << "Received large catalog update(>100mb): "
               :                      << item.key << " is "
               :                      << PrettyPrinter::Print(len, TUnit::BYTES);
               :       }
> Why print a warning at 100MB? What action would the user expect to take on 
Bharath said this is for supportability so we can take action on logs. I just made up 100mb,. that can be anything.


PS17, Line 1273: // Refresh the lib cache entries of any added functions and data sources
               :       if (catalog_object.type == TCatalogObjectType::FUNCTION) {
               :         DCHECK(catalog_object.__isset.fn);
               :         LibCache::instance()->SetNeedsRefresh(catalog_object.fn.hdfs_location);
               :       }
               :       if (catalog_object.type == TCatalogObjectType::DATA_SOURCE) {
               :         DCHECK(catalog_object.__isset.data_source);
               :         LibCache::instance()->SetNeedsRefresh(catalog_object.data_source.hdfs_location);
               :       }
> Can you leave a TODO saying that the frontend should return this informatio
Could you elaborate? what exactly is left to be done in TODO?


PS17, Line 1292: incremental_request
> I think a new variable is warranted here, e.g.:
I think it is better leave it this way. I think of the 1:1 correspondence between frontend and backend. this incremental_request is the same as incrementalRequest in frontend.


PS17, Line 1429: vector<TUpdateCatalogCacheRequest>(1, update_req)
> nit: I think you can just write {update_req} as a list initializer.
Done


http://gerrit.cloudera.org:8080/#/c/3067/17/fe/src/main/java/com/cloudera/impala/service/JniFrontend.java
File fe/src/main/java/com/cloudera/impala/service/JniFrontend.java:

PS17, Line 164:  
> nit: no space before :
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/3067
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I176db25124a32944f2396ce8aafbed49cac95928
Gerrit-PatchSet: 17
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hxu@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hxu@cloudera.com>
Gerrit-HasComments: Yes