geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Stolz <mst...@pivotal.io>
Subject Re: Adding parallel import/export of snapshots to gfsh
Date Tue, 22 Aug 2017 21:24:14 GMT
One other idea that hasn't been mentioned is making parallel the only way
for Partitioned Regions, and having --file serve the purpose of defining
both a path and a filename pattern where the bucket ID or whatever we're
using gets automatically inserted before the .gfd extension.

No need for a new option (--parallel).
No need for a new option (--path).

In fact, no need for a change to gfsh command at all.


--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Tue, Aug 22, 2017 at 2:15 PM, Nick Reich <nreich@pivotal.io> wrote:

> Parallel export will write the data to files on the bucket primary for each
> bucket, distributing the work (and therefore files) to all the members.
> That would be a big enough deviation from the current behavior (single file
> on single machine), that I think it makes it worth having the additional
> options (but I agree: less options is generally better).
>
> On Tue, Aug 22, 2017 at 1:59 PM, Jacob Barrett <jbarrett@pivotal.io>
> wrote:
>
> > On Tue, Aug 22, 2017 at 1:49 PM Nick Reich <nreich@pivotal.io> wrote:
> >
> > > The idea of deprecating —file in favor of path is interesting. I wonder
> > if
> > > instead of making them mutually exclusive to start, having —path be
> able
> > to
> > > support both modes from the start would be better? That way —file could
> > > still be used for the existing mode, but —path could be used instead
> (and
> > > override —file is both given?): that would provide a clear path forward
> > for
> > > how the command should be used, while fully supporting existing
> > workflows.
> > >
> >
> > This is what I meant by deprecating. Maybe even providing a message that
> if
> > --file is set that it is deprecated for --path.
> >
> >
> > > We need to continue to support both modes, as only Partitioned Regions
> > can
> > > make use of parallel export (it is parallelized on a bucket level).
> > >
> >
> > Ok, so why not just make parallel the only mode for partitioned. Then you
> > remove the need for --parallel and --path would work for any region,
> > non-partitioned would create a single file at that path and partitioned
> > would create several? I am all for less options. ;)
> >
> > -Jake
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message