Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
From: jeff saremi <jeffsaremi@hotmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: Re: Need guidance on Custom Compaction Policy
Thread-Topic: Need guidance on Custom Compaction Policy
Thread-Index: AQHSo0I3C/cTGN1ngEWHC9LnJOowm6GhW18AgAAG3Jk=
Date: Wed, 22 Mar 2017 21:39:09 +0000
Message-ID: <MWHPR14MB12935DCB23B7D9C3851FDCEEC13C0@MWHPR14MB1293.namprd14.prod.outlook.com>
References: <MWHPR14MB1293D533A1613E9F73A31DB6C13C0@MWHPR14MB1293.namprd14.prod.outlook.com>,<CALte62zWKU+Qx3+Lf3ywpbhKbuyJy6O3=ZYYn5J+Yf0CLTYbDg@mail.gmail.com>
In-Reply-To: <CALte62zWKU+Qx3+Lf3ywpbhKbuyJy6O3=ZYYn5J+Yf0CLTYbDg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [pYeRyweZS7/zVKi0behZyrs/9H/gB1Pe]
x-microsoft-exchange-diagnostics: 1;SN1NAM04HT147;5:VijoyBXC/CJfMW1FbUZffWxulolYAintlZHL2Ik8YJPU81FRkl9aKgq34jgOBUhQHwoF3EmowX/aHLgCmNuHSmtE0SxX3p9Y6jrSe6hFnz4D/sGfuOBiBqGhNLQOjYGqMmjEOeNU1rUD0xcePdEtvA==;24:W170Ts8f4XUrY78/6d9KgStNiXimh3ekDZuyhyyh4v10tLpgXYyu769qp7Twe4P9TGUBRWFgocGNRWxAFr5kymrDW7dgevL/1XGUH+BqXQk=;7:60AeRFFEpBDhhcEikvEiJbgMmtc03lMY8N3SmCwfeV+/Eqw+FuzuyYfNClgVG5iCRscTLeopvdpIDNcxjEUq94pZjjS1NXzqW/jveLxVhWC1PZZwjv/ZcvV4dgXy1m3sLFJx/EosQ5p/5jVXi1jH7Q0SsA4oj5TeizJCACCK8VRDXlVx9765tZpEtvitI3yD8Cs9Elo8i/Cy5DUuwjasH5e5mq+Y4zdksWZLnBJQCh005LL6+mrcGN4CjVsP5Nila85TuTAbMf6qLOQNEXTP7fAtROvY2gQrTwEzjxl+XfJcxyXLWyj53GZOyeKleTzP
x-incomingheadercount: 42
x-eopattributedmessage: 0
x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(10019020)(98900017);DIR:OUT;SFP:1102;SCL:1;SRVR:SN1NAM04HT147;H:MWHPR14MB1293.namprd14.prod.outlook.com;FPR:;SPF:None;LANG:en;
x-ms-office365-filtering-correlation-id: eb8be46a-3fe7-4c5e-b26a-08d4716be000
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201702061074)(5061506573)(5061507331)(1603103135)(1603101448)(1601125374)(1701031045);SRVR:SN1NAM04HT147;
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(432015281)(444000031);SRVR:SN1NAM04HT147;BCL:0;PCL:0;RULEID:;SRVR:SN1NAM04HT147;
x-forefront-prvs: 02543CD7CD
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_MWHPR14MB12935DCB23B7D9C3851FDCEEC13C0MWHPR14MB1293namp_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Mar 2017 21:39:09.8032
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1NAM04HT147
X-OriginalArrivalTime: 22 Mar 2017 21:39:11.0012 (UTC) FILETIME=[BE293E40:01D2A354]
archived-at: Wed, 22 Mar 2017 21:39:42 -0000

--_000_MWHPR14MB12935DCB23B7D9C3851FDCEEC13C0MWHPR14MB1293namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

@Ted
No i had not looked at DateTiered. I will. Thanks

@Vladimir
I will look at the example code you gave me. thanks


________________________________
From: Ted Yu <yuzhihong@gmail.com>
Sent: Wednesday, March 22, 2017 2:12:00 PM
To: user@hbase.apache.org
Subject: Re: Need guidance on Custom Compaction Policy

Have you taken look at http://hbase.apache.org/book.html#ops.date.tiered ?

Cheers

On Wed, Mar 22, 2017 at 12:29 PM, jeff saremi <jeffsaremi@hotmail.com>
wrote:

> I mentioned some of this in another thread. We have a readonly database
> which get bulk loaded using HFiles.
> We want to keep only two versions/generations of data. Since the size of
> data is massive we need to delete the older generation.
>
> Since we write one single HBase for each region for each CF for each
> generation, could we just yank the older files using a separate standalon=
e
> process? (sounds a little scary)
>
> If not, could we write a custom compactor? what's involved (some pointers
> please)? thanks
>
> Jeff
>

--_000_MWHPR14MB12935DCB23B7D9C3851FDCEEC13C0MWHPR14MB1293namp_--