hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5978) Create a tool to take fsimage and expose read-only WebHDFS API
Date Tue, 11 Mar 2014 22:43:43 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931082#comment-13931082

Haohui Mai commented on HDFS-5978:

   echo "  oiv                  apply the offline fsimage viewer to an fsimage"
+  echo "  wiv                  run the web fsimage viewer to an fsimage"

It might be more better to put this functionality as a subtool of the offline image viewer,
since it is intended to be a successor of the lsr tool.

It might be cleaner to separate the logics of loading fsimage and handling webhdfs requests
in {{FSImageHandler}}. You can wrap the information into a private static class:

+  private String[] stringTable;
+  private HashMap<Long, INode> inodes = Maps.newHashMap();
+  private HashMap<Long, long[]> dirmap = Maps.newHashMap();
+  private ArrayList<INodeReferenceSection.INodeReference> refList =
+      Lists.newArrayList();

+    public ChannelPipeline getPipeline() throws Exception {
+      ChannelPipeline pipeline = Channels.pipeline();
+      pipeline.addLast("httpDecoder", new HttpRequestDecoder());
+      pipeline.addLast("requestHandler", new FSImageHandler(inputFile));
+      pipeline.addLast("stringEncoder", new StringEncoder());
+      pipeline.addLast("httpEncoder", new HttpResponseEncoder());
+      return pipeline;
+    }
+  }

You might be able to create a static pipeline instead of a pipeline factory. See {{setPipeline()}}
for more details. I'm also unclear why {{StringEncoder}} is required.


+  public void testWebImageViewer() throws IOException, InterruptedException {
+    final String port = "9001";


The command line needs to accept both the listening host and the port. Otherwise it'll listen
to all interfaces by default. In unit test, it is also important to configure the port to
zero to avoid intermediate failures.

+      // wait until the viewer starts
+      Thread.sleep(3000);

You can use a condition variable here instead of sleeping.

+      HttpClient client = new DefaultHttpClient();
+      HttpGet httpGet =
+          new HttpGet("http://localhost:" + port + "/?op=LISTSTATUS");
+      HttpResponse response = client.execute(httpGet);
+      assertEquals(200, response.getStatusLine().getStatusCode());
+      assertEquals("application/json",
+                   response.getEntity().getContentType().getValue());

Using the built-in {{HttpUrlConnection}} is sufficient. It reduces the dependency of the unit

+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;

Please use jackson instead of gson. hadoop-hdfs does not depend on gson at all. That way there
is no additional dependency.

> Create a tool to take fsimage and expose read-only WebHDFS API
> --------------------------------------------------------------
>                 Key: HDFS-5978
>                 URL: https://issues.apache.org/jira/browse/HDFS-5978
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: tools
>            Reporter: Akira AJISAKA
>            Assignee: Akira AJISAKA
>              Labels: newbie
>         Attachments: HDFS-5978.patch
> Suggested in HDFS-5975.
> Add an option to exposes the read-only version of WebHDFS API for OfflineImageViewer.
You can imagine it looks very similar to jhat.
> That way we can allow the operator to use the existing command-line tool, or even the
web UI to debug the fsimage. It also allows the operator to interactively browsing the file
system, figuring out what goes wrong.

This message was sent by Atlassian JIRA

View raw message