Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
From: jeff saremi <jeffsaremi@hotmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: Re: Writing/Importing large number of records into HBase
Thread-Topic: Writing/Importing large number of records into HBase
Thread-Index: AQHSeRYiuvAq0wgJkkmsJr+a6RX1wqFNPqYAgAD/2vM=
Date: Sat, 28 Jan 2017 18:57:33 +0000
Message-ID: <CY4PR14MB13527DE04CA6795CFC933507C1490@CY4PR14MB1352.namprd14.prod.outlook.com>
References: <CY4PR14MB13524CF64D553A353674F5BBC1490@CY4PR14MB1352.namprd14.prod.outlook.com>,<CALte62xQ=7XAgn0D18T5NA1-H16zf_J9-qdTANs=qjhQMq7XFg@mail.gmail.com>
In-Reply-To: <CALte62xQ=7XAgn0D18T5NA1-H16zf_J9-qdTANs=qjhQMq7XFg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [dBDY1T28ep41tx1QHKuUc5qF275bbsE8]
x-incomingheadercount: 39
x-eopattributedmessage: 0
x-microsoft-exchange-diagnostics: 1;CO1NAM03HT048;7:yTztL+hLIopB++XdeAh38+PKAQyaaxIcvuWtJuAFBGm5e/Iol002K8S84//fwhv4t6RmGclPNL1kDvBOLHFM7dioCaE2XHbxF+eu+wEADdBkonZD3uCM05oN3PThO3SlqP8/Wbbizt0J8kfdPprNPu55P7+JlQZ1TznQYbO/GFnvdNskEJN0Y0BfdhDNoav/fD5enRouIcF5plz/fY5Gt5NT2uVWGtMqCNny2KCPe7B8gQ2Md6+X5RWbp6yx5r7WbFwARxSq2y7WNoElxdZelr5JnI4Rileu6XUIjgG+xDRTDfj81r+TXoKgHMCX8al16Wa7idTpJD3txYmisBCennHlrh1KMbzsYyjraWXXOKvMcdX7L7JCLOqDke1hfB7n9eHR7fZXvIAhZpFT0wAc8tesUxwJgrOGOau7b7dDHCN0nMLLleLzvGeE+hKkmF+w
x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(10019020)(98900005);DIR:OUT;SFP:1102;SCL:1;SRVR:CO1NAM03HT048;H:CY4PR14MB1352.namprd14.prod.outlook.com;FPR:;SPF:None;LANG:en;
x-ms-office365-filtering-correlation-id: 202e43b0-a987-49fd-5d94-08d447af84d2
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(1601124038)(5061506344)(5061507293)(1603103113)(1601125047)(1603101340)(1701031023);SRVR:CO1NAM03HT048;
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(444111334)(444112120)(432015012)(82015046);SRVR:CO1NAM03HT048;BCL:0;PCL:0;RULEID:;SRVR:CO1NAM03HT048;
x-forefront-prvs: 02015246A9
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_CY4PR14MB13527DE04CA6795CFC933507C1490CY4PR14MB1352namp_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jan 2017 18:57:33.8219
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1NAM03HT048
X-OriginalArrivalTime: 28 Jan 2017 18:57:36.0267 (UTC) FILETIME=[63BFCDB0:01D27998]
archived-at: Sat, 28 Jan 2017 18:57:47 -0000

--_000_CY4PR14MB13527DE04CA6795CFC933507C1490CY4PR14MB1352namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

No iI had not.I will take a look. Thanks Ted


________________________________
From: Ted Yu <yuzhihong@gmail.com>
Sent: Friday, January 27, 2017 7:41 PM
To: user@hbase.apache.org
Subject: Re: Writing/Importing large number of records into HBase

Have you looked at hbase-spark module (currently in master branch) ?

See hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/dataso=
urces/AvroSource.scala
and hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/DefaultSourceS=
uite.scala
for examples.

There may be other options.

FYI

On Fri, Jan 27, 2017 at 7:28 PM, jeff saremi <jeffsaremi@hotmail.com> wrote=
:

> Hi
> I'm seeking some pointers/guidance on what we could do to insert billions
> of records that we already have in avro files in hadoop into HBase.
>
> I read some articles online and one of them recommended using HFile
> format. I took a cursory look at the documentation for that. Given the
> complexity of that I think that may be the last resort we want to pursue.
> Unless some library is out there that easily helps us write our files int=
o
> that format. I didn't see any.
> Assuming that the Hbase native client may be our best bet, is there any
> advice around pre-paritioning our records or such techniques that we coul=
d
> use?
> thanks
>
> Jeff
>

--_000_CY4PR14MB13527DE04CA6795CFC933507C1490CY4PR14MB1352namp_--