impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Huaisi Xu (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-3499: Split catalog update
Date Fri, 10 Jun 2016 21:27:02 GMT
Huaisi Xu has posted comments on this change.

Change subject: IMPALA-3499: Split catalog update
......................................................................


Patch Set 17:

(3 comments)

> > No. I just used 10k as a limit. I do not have resources to test
 > 2gb metadata.
 > 
 > How can we get confidence that this avoids the bug w/o testing it
 > at the 2GB limit? Bharath mentioned that we might have access to
 > metadata dumps of that size - could you check to see if that's
 > true?

I will ping him to see how to do that.

http://gerrit.cloudera.org:8080/#/c/3067/17/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

PS17, Line 1255: if (len > 100 * 1024 * 1024 /* 100MB */) {
               :         LOG(WARNING) << "Received large catalog update(>100mb):
"
               :                      << item.key << " is "
               :                      << PrettyPrinter::Print(len, TUnit::BYTES);
               :       }
> Bharath, can you help me understand how this is actionable (either for supp
I think he was thinking about if there some particular large update then support can ask the
customer to take care of those. Not exactly sure though.


PS17, Line 1273: // Refresh the lib cache entries of any added functions and data sources
               :       if (catalog_object.type == TCatalogObjectType::FUNCTION) {
               :         DCHECK(catalog_object.__isset.fn);
               :         LibCache::instance()->SetNeedsRefresh(catalog_object.fn.hdfs_location);
               :       }
               :       if (catalog_object.type == TCatalogObjectType::DATA_SOURCE) {
               :         DCHECK(catalog_object.__isset.data_source);
               :         LibCache::instance()->SetNeedsRefresh(catalog_object.data_source.hdfs_location);
               :       }
> The frontend should return the list of functions and data sources in an upd
got it!.. done.. but not sure how this can be easily done/beneficial under our current structure..


PS17, Line 1292: incremental_request
> I would prefer you changed it, please. The smaller scope that you can const
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/3067
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I176db25124a32944f2396ce8aafbed49cac95928
Gerrit-PatchSet: 17
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hxu@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hxu@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message