drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edmon Begoli <ebeg...@gmail.com>
Subject Re: The meaning of the methods in StoragePlugin and EasyFormatPlugin
Date Sun, 30 Aug 2015 13:52:23 GMT
Thanks, Jacques. I might add your answers in a Javadoc and also maybe write
an addendum to the documentation as I just told Ted.

It helps me learn and it will benefit others.

On Saturday, August 29, 2015, Jacques Nadeau <jacques@dremio.com> wrote:

> supportsRead -> Drill can read a table from this format (e.g. read a json
> file)
> supportsWrite -> Drill can write a table to this format (e.g. write a json
> file)
>
> question: -- what does StoragePluginOptimizerRule do?
>   public Set<StoragePluginOptimizerRule>
> getOptimizerRules(OptimizerRulesContext optimizerContext);
>
> You can generally return an empty set for this.  You could possibly expose
> one or more specialized rules for this format that the Drill query
> optimizer would leverage. For example, if your plugin supported built-in
> aggregations, you could expose a rule that rewrote a aggregate < your_scan
> to be simply a your_scan_with_aggregation.
>
>   public boolean isBlockSplittable()
> Whether or not your format can be arbitrarily split and handle a random
> start point.  For example, a csv file supports this since our reader can
> start in the middle of a line and stop at the next line beyond a given
> split point.  ON the flipside, Parquet can't use arbitrary split points.
> JSON also doesn't support arbitrary split points since (in the most general
> case), there is no way to search to the start of the next record.
>
> question: -- what criteria makes this truth; What does this mean?
>   public boolean isCompressible()
>
> Whether this format could also be in a compression container.  For example:
> csv.gz versus csv.  This is an external compression container that would be
> handled outside of your format code. Parquet also wouldn't support this
> since it has an internal compression scheme.
>
>
> question: -- is this configuration that is specified in the "Storage"
> section on the web ui/config. files?
>   public StoragePluginConfig getConfig();
>
> Yes, should be a jackson serializable object that extends
> StoragePluginConfig.  What is input when configuring in the web ui.
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Sat, Aug 29, 2015 at 7:01 PM, Edmon Begoli <ebegoli@gmail.com
> <javascript:;>> wrote:
>
> > On *StoragePlugin*:
> >
> > question: -- I assume this means that sources of this storage format kind
> > can be read from the original sources, correct?
> >  public boolean supportsRead();
> >
> > question: -- Is this writing to the original source or does this mean
> > creation of the table for this format?
> >   public boolean supportsWrite();
> >
> > question: -- what does StoragePluginOptimizerRule do?
> >   public Set<StoragePluginOptimizerRule>
> > getOptimizerRules(OptimizerRulesContext optimizerContext);
> >
> > question: -- is this configuration provided on the web ui or config.
> files?
> >   public StoragePluginConfig getConfig();
> >
> >
> > On *EasyFormatPlugin*:
> >
> > question: -- what criteria makes this truth; What does this mean?
> >   public boolean isBlockSplittable()
> >
> > question: -- what criteria makes this truth; What does this mean?
> >   public boolean isCompressible()
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message