hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From slim tebourbi <slimtbou...@gmail.com>
Subject Re: When creating and loading HFile programmatically to HBase new entries are unavailable
Date Tue, 20 Mar 2012 23:13:14 GMT
Hello,

see :
http://stackoverflow.com/questions/9665877/when-creating-and-loading-hfile-programmatically-to-hbase-new-entries-are-unavai

slim Tebourbi.

Le 12 mars 2012 12:12, Nicolas Thiébaud <nicolas@captaindash.com> a écrit :

> I'm trying to create HFiles programmatically and loading them in a running
> HBase instance. I found a lot of info in HFileOutputFormat and in
> LoadIncrementalHFiles
>
> I managed to create the new HFile, send it to the cluster. In the cluster
> web interface the new store file appears but the new keyrange is
> unavailable for query.
>
> InputStream stream =
> ProgrammaticHFileGeneration.class.getResourceAsStream("ga-hourly.txt");
> BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
> String line = null;
>  Map<byte[], String> rowValues = new HashMap<byte[], String>();
>  while((line = reader.readLine())!=null) {
> String[] vals = line.split(",");
> String row = new
>
> StringBuilder(vals[0]).append(".").append(vals[1]).append(".").append(vals[2]).append(".").append(vals[3]).toString();
> rowValues.put(row.getBytes(), line);
> }
>  List<byte[]> keys = new ArrayList<byte[]>(rowValues.keySet());
> Collections.sort(keys, byteArrComparator);
>  HBaseTestingUtility testingUtility = new HBaseTestingUtility();
> testingUtility.startMiniCluster();
>  testingUtility.createTable("table".getBytes(), "data".getBytes());
>  Writer writer = new HFile.Writer(testingUtility.getTestFileSystem(),
> new Path("/tmp/hfiles/data/hfile"),
> HFile.DEFAULT_BLOCKSIZE, Compression.Algorithm.NONE,
> KeyValue.KEY_COMPARATOR);
>  for(byte[] key:keys) {
> writer.append(new KeyValue(key, "data".getBytes(), "d".getBytes(),
> rowValues.get(key).getBytes()));
> }
>  writer.appendFileInfo(StoreFile.BULKLOAD_TIME_KEY,
> Bytes.toBytes(System.currentTimeMillis()));
> writer.appendFileInfo(StoreFile.MAJOR_COMPACTION_KEY, Bytes.toBytes(true));
> writer.close();
>  Configuration conf = testingUtility.getConfiguration();
>  LoadIncrementalHFiles loadTool = new LoadIncrementalHFiles(conf);
> HTable hTable = new HTable(conf, "table".getBytes());
>  loadTool.doBulkLoad(new Path("/tmp/hfiles"), hTable);
>  ResultScanner scanner = hTable.getScanner("data".getBytes());
> Result next = null;
> System.out.println("Scanning");
> while((next = scanner.next()) != null) {
> System.out.format("%s %s\n", new String(next.getRow()), new
> String(next.getValue("data".getBytes(), "d".getBytes())));
> }
>
> What am I doing wrong ? I managed running M/R to Hbase, but I really want
> the programmatic use case working..
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message