hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas ThiƩbaud <nico...@captaindash.com>
Subject When creating and loading HFile programmatically to HBase new entries are unavailable
Date Mon, 12 Mar 2012 11:12:07 GMT
I'm trying to create HFiles programmatically and loading them in a running
HBase instance. I found a lot of info in HFileOutputFormat and in
LoadIncrementalHFiles

I managed to create the new HFile, send it to the cluster. In the cluster
web interface the new store file appears but the new keyrange is
unavailable for query.

InputStream stream =
ProgrammaticHFileGeneration.class.getResourceAsStream("ga-hourly.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String line = null;
 Map<byte[], String> rowValues = new HashMap<byte[], String>();
 while((line = reader.readLine())!=null) {
String[] vals = line.split(",");
String row = new
StringBuilder(vals[0]).append(".").append(vals[1]).append(".").append(vals[2]).append(".").append(vals[3]).toString();
rowValues.put(row.getBytes(), line);
}
 List<byte[]> keys = new ArrayList<byte[]>(rowValues.keySet());
Collections.sort(keys, byteArrComparator);
 HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster();
 testingUtility.createTable("table".getBytes(), "data".getBytes());
 Writer writer = new HFile.Writer(testingUtility.getTestFileSystem(),
new Path("/tmp/hfiles/data/hfile"),
HFile.DEFAULT_BLOCKSIZE, Compression.Algorithm.NONE,
KeyValue.KEY_COMPARATOR);
 for(byte[] key:keys) {
writer.append(new KeyValue(key, "data".getBytes(), "d".getBytes(),
rowValues.get(key).getBytes()));
}
 writer.appendFileInfo(StoreFile.BULKLOAD_TIME_KEY,
Bytes.toBytes(System.currentTimeMillis()));
writer.appendFileInfo(StoreFile.MAJOR_COMPACTION_KEY, Bytes.toBytes(true));
writer.close();
 Configuration conf = testingUtility.getConfiguration();
 LoadIncrementalHFiles loadTool = new LoadIncrementalHFiles(conf);
HTable hTable = new HTable(conf, "table".getBytes());
 loadTool.doBulkLoad(new Path("/tmp/hfiles"), hTable);
 ResultScanner scanner = hTable.getScanner("data".getBytes());
Result next = null;
System.out.println("Scanning");
while((next = scanner.next()) != null) {
System.out.format("%s %s\n", new String(next.getRow()), new
String(next.getValue("data".getBytes(), "d".getBytes())));
}

What am I doing wrong ? I managed running M/R to Hbase, but I really want
the programmatic use case working..

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message