hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Surendra , Manchikanti" <surendra.manchika...@gmail.com>
Subject Re: Loading text files from local file system
Date Wed, 17 Apr 2013 05:19:32 GMT
Hi Joshi,

You can use Flume + AsyncHbaseSink / HBasesink to move data from local file
sytem to HBase.



Thanks,
Surendra M

-- Surendra Manchikanti


On Wed, Apr 17, 2013 at 10:01 AM, Omkar Joshi
<Omkar.Joshi@lntinfotech.com>wrote:

> The background thread is here :
>
>
> http://mail-archives.apache.org/mod_mbox/hbase-user/201304.mbox/%3CE689A42B73C5A545AD77332A4FC75D8C1EFBE84153@VSHINMSMBX01.vshodc.lntinfotech.com%3E
>
> Following are the commands that I'm using to load files onto HBase :
>
> HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath`
> ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-0.94.6.1.jar importtsv
> '-Dimporttsv.separator=;'
> -Dimporttsv.columns=HBASE_ROW_KEY,PRODUCT_INFO:NAME,PRODUCT_INFO:CATEGORY,PRODUCT_INFO:GROUP,PRODUCT_INFO:COMPANY,PRODUCT_INFO:COST,PRODUCT_INFO:COLOR,PRODUCT_INFO:BLANK_COLUMN
> -Dimporttsv.bulk.output=hdfs://cldx-1139-1033:9000/hbase/storefileoutput_6
> PRODUCTS hdfs://cldx-1139-1033:9000/hbase/copiedFromLocal/product_6.txt
>
> HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath`
> ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-0.94.6.1.jar
> completebulkload hdfs://cldx-1139-1033:9000/hbase/storefileoutput_6 PRODUCTS
>
> As seen, the text files to be loaded in HBase first need to be loaded on
> HDFS. Given our infrastructure constraints/limitations, I'm getting space
> issues. The data in the text files is around 20GB + replication is
> consuming a lot of DFS.
>
> Is there a way wherein a text file can be loaded directly from the local
> file system onto HBase?
>
> Regards,
> Omkar Joshi
>
> ________________________________
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message