Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
Date: Mon, 12 Mar 2012 12:12:07 +0100
Message-ID: 
 <CAFWP4dGpc8AFD3aVPVAwimyfkk+e6GZkMdG9tJCZB0k2RMDKBw@mail.gmail.com>
Subject: When creating and loading HFile programmatically to HBase new entries
 are unavailable
From: =?ISO-8859-1?Q?Nicolas_Thi=E9baud?= <nicolas@captaindash.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=f46d044784971880db04bb09d0e4

--f46d044784971880db04bb09d0e4
Content-Type: text/plain; charset=ISO-8859-1

I'm trying to create HFiles programmatically and loading them in a running
HBase instance. I found a lot of info in HFileOutputFormat and in
LoadIncrementalHFiles

I managed to create the new HFile, send it to the cluster. In the cluster
web interface the new store file appears but the new keyrange is
unavailable for query.

InputStream stream =
ProgrammaticHFileGeneration.class.getResourceAsStream("ga-hourly.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String line = null;
 Map<byte[], String> rowValues = new HashMap<byte[], String>();
 while((line = reader.readLine())!=null) {
String[] vals = line.split(",");
String row = new
StringBuilder(vals[0]).append(".").append(vals[1]).append(".").append(vals[2]).append(".").append(vals[3]).toString();
rowValues.put(row.getBytes(), line);
}
 List<byte[]> keys = new ArrayList<byte[]>(rowValues.keySet());
Collections.sort(keys, byteArrComparator);
 HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster();
 testingUtility.createTable("table".getBytes(), "data".getBytes());
 Writer writer = new HFile.Writer(testingUtility.getTestFileSystem(),
new Path("/tmp/hfiles/data/hfile"),
HFile.DEFAULT_BLOCKSIZE, Compression.Algorithm.NONE,
KeyValue.KEY_COMPARATOR);
 for(byte[] key:keys) {
writer.append(new KeyValue(key, "data".getBytes(), "d".getBytes(),
rowValues.get(key).getBytes()));
}
 writer.appendFileInfo(StoreFile.BULKLOAD_TIME_KEY,
Bytes.toBytes(System.currentTimeMillis()));
writer.appendFileInfo(StoreFile.MAJOR_COMPACTION_KEY, Bytes.toBytes(true));
writer.close();
 Configuration conf = testingUtility.getConfiguration();
 LoadIncrementalHFiles loadTool = new LoadIncrementalHFiles(conf);
HTable hTable = new HTable(conf, "table".getBytes());
 loadTool.doBulkLoad(new Path("/tmp/hfiles"), hTable);
 ResultScanner scanner = hTable.getScanner("data".getBytes());
Result next = null;
System.out.println("Scanning");
while((next = scanner.next()) != null) {
System.out.format("%s %s\n", new String(next.getRow()), new
String(next.getValue("data".getBytes(), "d".getBytes())));
}

What am I doing wrong ? I managed running M/R to Hbase, but I really want
the programmatic use case working..

--f46d044784971880db04bb09d0e4--