drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clifford Resnick <cresn...@mediamath.com>
Subject Question about Kudu Storage and storages in general
Date Thu, 20 Jul 2017 16:49:23 GMT
I’m trying out the Kudu Storage with Kudu 1.4 without success, always getting a kudu error
-> “Invalid scan start key: Error decoding composite key component ‘my-key': key too
short: <redacted>”

I know this storage is experimental, but I’m hoping to get it to work. Looking at the code
I noticed it’s based on a deprecated way of locating Kudu Tablets for scans, doing a general
location mapping of Tablet Servers, I suppose to better colocate Drillbit scans. Instead of
this, Kudu now recommends using the KuduScanToken api:

" A scan token describes a partial scan of a Kudu table limited to a single
 contiguous physical location. Using the {@link KuduScanTokenBuilder}, clients can
 describe the desired scan, including predicates, bounds, timestamps, and
 caching, and receive back a collection of scan tokens.

 Each scan token may be separately turned into a scanner using
 {@link #intoScanner}, with each scanner responsible for a disjoint section
 of the table.

 Scan tokens may be serialized using the {@link #serialize} method and
 deserialized back into a scanner using the {@link #deserializeIntoScanner}
 method. This allows use cases such as generating scan tokens in the planner
 component of a query engine, then sending the tokens to execution nodes based
 on locality, and then instantiating the scanners on those nodes.

 Scan token locality information can be inspected using the {@link #getTablet}
 method.”

I’m new to Drill, but it seemed to me that this api could be retro-fitted to good effect
into KuduGroupScan, but I didn’t get very far given that I couldn’t even suss out where
the predicates were in the drill code. Unless I completely misunderstand the concept, it seems
the Kudu Storage must be pushing the predicate to lower code levels and is therefore not exposed
to them. If you read the above, my hope is that there is a way to serialize the Kudu ScanTokens
to Drillbits to be used as scanners. Does anyone know if this is possible using the Drill
execution path? If so can someone please point me to documentation/examples/tests I can consult
to help clarify my muddled understanding of Drill?

-Cliff




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message