Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
From: jeff saremi <jeffsaremi@hotmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: Re: Writing/Importing large number of records into HBase
Thread-Topic: Writing/Importing large number of records into HBase
Thread-Index: AQHSeRYiuvAq0wgJkkmsJr+a6RX1wqFNPqYAgAABnYCAAALwgIAABO4AgAD2EoM=
Date: Sat, 28 Jan 2017 18:56:18 +0000
Message-ID: <CY4PR14MB1352E5473567F918E768D03DC1490@CY4PR14MB1352.namprd14.prod.outlook.com>
References: <CY4PR14MB13524CF64D553A353674F5BBC1490@CY4PR14MB1352.namprd14.prod.outlook.com>
 <CALte62xQ=7XAgn0D18T5NA1-H16zf_J9-qdTANs=qjhQMq7XFg@mail.gmail.com>
 <CA+MEsbBv48z9F+C0FtwUjU-Q_uKQ3G=k_MyrMPA63cr6S105hA@mail.gmail.com>
 <43F4BA97-4287-49A0-AF8C-8BFF93A754C8@gmail.com>,<CA+MEsbBrFfQsNGVx5CA5OSHY6C9L=A3Pz-QXzdExc9TBWVGC4Q@mail.gmail.com>
In-Reply-To: <CA+MEsbBrFfQsNGVx5CA5OSHY6C9L=A3Pz-QXzdExc9TBWVGC4Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [vTxXCMSX9KUizm+Sb+EI/g9gkPrHM9df]
x-incomingheadercount: 39
x-eopattributedmessage: 0
x-microsoft-exchange-diagnostics: 1;CO1NAM03HT216;7:i6ekgoHO2dNatv2NG9PsMiEhRMhhvuObCXQ/JNfRdl11wN5RS2c0p4fCopW+KN33dYdBzNmI7EmpDd7wrpyJgXEPsgMAoqs12SHHkPMnpOfYJ3CyHP9vglMoIi/LJ1xIj6pcPPVV6f1Qhm+fo4GwMWhYLkyx6hpfDAPA2up1jTuk4Kjt/BMcSemX427DCAAiWdDUUAaUUoiqX5I+6+iBY9qKOETpEu48MZv+ZuLIoREL755S6HGhC43ee4XQmqyogIdWGPMKTqVnF3y5/nc7/vd6DbpxeRn8B5VQRkzaMAy4XNEOb4F3Dqcjz49tVB7VJNak3Egr6kJ5j7TomUyS6V+MYsMhYFq2LX4O3TQzmZydw7z4lLZjQ1LikCLr4O4ERH4O0UPl1Tf56RoMCKiSmqhl4Xe1vzbkS3/pI4Wk8yALUqDUssSXAyghswEXQwVgQYQtHSYU8K7g55GGvmV0Ew==
x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(10019020)(98900005);DIR:OUT;SFP:1102;SCL:1;SRVR:CO1NAM03HT216;H:CY4PR14MB1352.namprd14.prod.outlook.com;FPR:;SPF:None;LANG:en;
x-ms-office365-filtering-correlation-id: 71a806f4-65f5-4e33-ec2c-08d447af57f6
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(1601124038)(5061506344)(5061507270)(1603103113)(1601125047)(1603101340)(1701031023);SRVR:CO1NAM03HT216;
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(444111334)(444112120)(432015012)(82015046);SRVR:CO1NAM03HT216;BCL:0;PCL:0;RULEID:;SRVR:CO1NAM03HT216;
x-forefront-prvs: 02015246A9
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_CY4PR14MB1352E5473567F918E768D03DC1490CY4PR14MB1352namp_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jan 2017 18:56:18.5690
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1NAM03HT216
X-OriginalArrivalTime: 28 Jan 2017 18:56:20.0634 (UTC) FILETIME=[36AB1FA0:01D27998]
archived-at: Sat, 28 Jan 2017 18:57:16 -0000

--_000_CY4PR14MB1352E5473567F918E768D03DC1490CY4PR14MB1352namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Thank you Chetan


________________________________
From: Chetan Khatri <chetan.opensource@gmail.com>
Sent: Friday, January 27, 2017 8:15 PM
To: user@hbase.apache.org
Subject: Re: Writing/Importing large number of records into HBase

Oh. Sorry.
https://github.com/apache/hbase/blob/master/hbase-spark/src/main/java/org/a=
pache/hadoop/hbase/spark/example/hbasecontext/JavaHBaseBulkPutExample.java
[https://avatars1.githubusercontent.com/u/47359?v=3D3&s=3D400]<https://gith=
ub.com/apache/hbase/blob/master/hbase-spark/src/main/java/org/apache/hadoop=
/hbase/spark/example/hbasecontext/JavaHBaseBulkPutExample.java>

hbase/JavaHBaseBulkPutExample.java at master =B7 apache ...<https://github.=
com/apache/hbase/blob/master/hbase-spark/src/main/java/org/apache/hadoop/hb=
ase/spark/example/hbasecontext/JavaHBaseBulkPutExample.java>
github.com
hbase - Mirror of Apache HBase ... Switch branches/tags. Branches; Tags


On Sat, Jan 28, 2017 at 9:27 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Chetan:
> The link you posted was from personal repo.
>
> There hasn't been commit for at least a year.
>
> Meanwhile, the hbase-spark module in hbase repo is being actively
> maintained.
>
> FYI
>
> > On Jan 27, 2017, at 7:47 PM, Chetan Khatri <chetan.opensource@gmail.com=
>
> wrote:
> >
> > Adding to @Ted Check Bulk Put Example -
> > https://github.com/tmalaska/SparkOnHBase/blob/master/src/
[https://avatars3.githubusercontent.com/u/1946016?v=3D3&s=3D400]<https://gi=
thub.com/tmalaska/SparkOnHBase/blob/master/src/>

tmalaska/SparkOnHBase<https://github.com/tmalaska/SparkOnHBase/blob/master/=
src/>
github.com
Contribute to SparkOnHBase development by creating an account on GitHub.


> main/scala/org/apache/hadoop/hbase/spark/example/hbasecontext/
> HBaseBulkPutExampleFromFile.scala
> >
> >> On Sat, Jan 28, 2017 at 9:11 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>
> >> Have you looked at hbase-spark module (currently in master branch) ?
> >>
> >> See hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/
> >> example/datasources/AvroSource.scala
> >> and hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/
> >> DefaultSourceSuite.scala
> >> for examples.
> >>
> >> There may be other options.
> >>
> >> FYI
> >>
> >> On Fri, Jan 27, 2017 at 7:28 PM, jeff saremi <jeffsaremi@hotmail.com>
> >> wrote:
> >>
> >>> Hi
> >>> I'm seeking some pointers/guidance on what we could do to insert
> billions
> >>> of records that we already have in avro files in hadoop into HBase.
> >>>
> >>> I read some articles online and one of them recommended using HFile
> >>> format. I took a cursory look at the documentation for that. Given th=
e
> >>> complexity of that I think that may be the last resort we want to
> pursue.
> >>> Unless some library is out there that easily helps us write our files
> >> into
> >>> that format. I didn't see any.
> >>> Assuming that the Hbase native client may be our best bet, is there a=
ny
> >>> advice around pre-paritioning our records or such techniques that we
> >> could
> >>> use?
> >>> thanks
> >>>
> >>> Jeff
> >>
>

--_000_CY4PR14MB1352E5473567F918E768D03DC1490CY4PR14MB1352namp_--