accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher <ctubb...@apache.org>
Subject Re: Accumulo on s3
Date Thu, 12 May 2016 18:08:01 GMT
getWalDirs() is used only in legacy code intended to ease upgrades from
1.4.x to 1.5.x. It probably has no value in the current code, as I'm not
sure we've even tested upgrades from 1.4.x to 1.7.x.

On Thu, May 12, 2016 at 11:03 AM Shawn Walker <accumulo@shawn-walker.net>
wrote:

> Teng Qiu,
>
> I don't believe your proposed change would suffice.  I haven't spent much
> time exploring how one might implement my suggestion.  Two things:
>
> 1. I've just recently started working with Accumulo; as such I'm not
> certain that WALs are the only files Accumulo uses that need more
> capability than S3 can provide.  Just that it should be reasonable to store
> RFiles in S3, modulo some concerns about renaming.
>
> 2. Changing getWalDirs() would not be effective, as this function isn't
> used everywhere that matters. Indeed, my (quick) skim of the source
> indicates that getWalDirs() isn't used anywhere that matters.  For example,
> the ultimate file location for a tserver's WAL looks to be decided at
> [tserver
> - DfsLogger.open(...):379]
> <
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java#L379
> >
> which doesn't call ServerConstants.getWalDirs().
>
> Instead, I was suggesting changing the signature of [server/base -
> VolumeManager.choose(...):161]
> <
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L161
> >
> to take an additional parameter indicating the purpose of the volume (RFile
> storage, WAL storage, temporary storage, etc), and passing that information
> along to the configured VolumeChooser by modifying [server/base -
> VolumeChooserEnvironment]
> <
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooserEnvironment.java
> >.
> The ultimate goal would be to allow a custom VolumeChooser to better choose
> volumes based on what type of use the volume would need to support.
>
> I'm not entirely certain this tactic would work, but it would be where I
> might start.
>
> --
> Shawn Walker
>
>
> On Tue, May 10, 2016 at 11:49 AM, chutium <tengqiu@gmail.com> wrote:
>
> > Hi Shawn, thanks for the detailed reply, we took a look into accumulo
> code,
> > and found the constant WAL_DIR defined in ServerConstants:
> >
> >
> >
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L139
> >
> >
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L151
> > (we are using 1.7.1)
> >
> > is that means, we can simply modify the return value of getWalDirs() to
> > change the location of WAL logs?
> >
> > Thanks
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-accumulo.1065345.n5.nabble.com/Accumulo-on-s3-tp16737p16872.html
> > Sent from the Developers mailing list archive at Nabble.com.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message