accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Walker <accum...@shawn-walker.net>
Subject Re: Accumulo on s3
Date Thu, 12 May 2016 15:03:28 GMT
Teng Qiu,

I don't believe your proposed change would suffice.  I haven't spent much
time exploring how one might implement my suggestion.  Two things:

1. I've just recently started working with Accumulo; as such I'm not
certain that WALs are the only files Accumulo uses that need more
capability than S3 can provide.  Just that it should be reasonable to store
RFiles in S3, modulo some concerns about renaming.

2. Changing getWalDirs() would not be effective, as this function isn't
used everywhere that matters. Indeed, my (quick) skim of the source
indicates that getWalDirs() isn't used anywhere that matters.  For example,
the ultimate file location for a tserver's WAL looks to be decided at [tserver
- DfsLogger.open(...):379]
<https://github.com/apache/accumulo/blob/rel/1.7.1/server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java#L379>
which doesn't call ServerConstants.getWalDirs().

Instead, I was suggesting changing the signature of [server/base -
VolumeManager.choose(...):161]
<https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L161>
to take an additional parameter indicating the purpose of the volume (RFile
storage, WAL storage, temporary storage, etc), and passing that information
along to the configured VolumeChooser by modifying [server/base -
VolumeChooserEnvironment]
<https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooserEnvironment.java>.
The ultimate goal would be to allow a custom VolumeChooser to better choose
volumes based on what type of use the volume would need to support.

I'm not entirely certain this tactic would work, but it would be where I
might start.

--
Shawn Walker


On Tue, May 10, 2016 at 11:49 AM, chutium <tengqiu@gmail.com> wrote:

> Hi Shawn, thanks for the detailed reply, we took a look into accumulo code,
> and found the constant WAL_DIR defined in ServerConstants:
>
>
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L139
>
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L151
> (we are using 1.7.1)
>
> is that means, we can simply modify the return value of getWalDirs() to
> change the location of WAL logs?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/Accumulo-on-s3-tp16737p16872.html
> Sent from the Developers mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message