From dev-return-2321-archive-asf-public=cust-asf.ponee.io@tephra.incubator.apache.org Wed Jan 23 03:10:26 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 63A78180634 for ; Wed, 23 Jan 2019 03:10:26 +0100 (CET) Received: (qmail 57548 invoked by uid 500); 23 Jan 2019 02:10:25 -0000 Mailing-List: contact dev-help@tephra.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@tephra.incubator.apache.org Delivered-To: mailing list dev@tephra.incubator.apache.org Received: (qmail 57516 invoked by uid 99); 23 Jan 2019 02:10:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2019 02:10:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 256B8C02B3 for ; Wed, 23 Jan 2019 02:10:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2 X-Spam-Level: ** X-Spam-Status: No, score=2 tagged_above=-999 required=6.31 tests=[HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id vAg4qKb4nedx for ; Wed, 23 Jan 2019 02:10:22 +0000 (UTC) Received: from mail-ot1-f47.google.com (mail-ot1-f47.google.com [209.85.210.47]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 409195FDE4 for ; Wed, 23 Jan 2019 02:10:22 +0000 (UTC) Received: by mail-ot1-f47.google.com with SMTP id u16so566048otk.8 for ; Tue, 22 Jan 2019 18:10:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=o2Hpae+Jq3pgvjILP46RgNEyv0ZUA4zm7nqf6xIQKic=; b=SmuAX6DGuN7fsAqvzjPiP9V2ovdthA50NdrnKjbzDS+Lc+rb8Mh0r1jh+ov42p6Qre FHII6AFtkcEYLJHrw4rd5Qgewychfgru8eyhRMScAGZX6JvUOenMCJbYiwG5DVUgF0tE 0Iz+JD8in3VUB1oRCIJv1rIlG9Twyd2li/TzR+wHz5/RuKy1gIm1gAXSEGsEajqLrL3k FhKJh32vGUZBDAUHV3Qc9ObUoBUqoTzozTkFwA9ILPW636EwjM7jHwBXDYclkQi1Jgso bHkoT5n03GnCDunb5h1rSIj+5iFUklpVNjjYpmGNHvIafjDvbr8xsH9j6+jM0SgVj/Do slfA== X-Gm-Message-State: AJcUukeQVcqup8zofRQPqzSfHYiGeZpOEw5iwFXygQJeR8EZvAlJQUyp fPcYx97XRRBGA7qZd/EiSpwFfmhSGPo0I1DfAGzFB2jN X-Google-Smtp-Source: ALg8bN4wh3QaJ3tnVBNoekZ1zUsPUhql4RGGHAJdRQNDE6PvZnrcFlQEcnyICp3zTWtqIdrGmaPZGdLOWAM+UkBuNuo= X-Received: by 2002:a9d:2c46:: with SMTP id f64mr251980otb.192.1548209420687; Tue, 22 Jan 2019 18:10:20 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Andreas Neumann Date: Tue, 22 Jan 2019 18:10:09 -0800 Message-ID: Subject: Re: Reading tephra snapshots To: dev@tephra.incubator.apache.org Content-Type: multipart/alternative; boundary="0000000000008edcc90580169811" --0000000000008edcc90580169811 Content-Type: text/plain; charset="UTF-8" Hi Varun, for debugging purposes, you should only need to read the last snapshot and the transaction log that follows it. Older snapshots should not be needed. Tephra does not have a CLI tool to do that yet, CDAP (which uses Tephra) has a similar tool, however a little more complex because it implements a snapshot server-side and the inspection of that snapshot client-side. See: - https://github.com/cdapio/cdap/blob/develop/cdap-data-fabric/src/main/java/co/cask/cdap/data2/transaction/TransactionManagerDebuggerMain.java - https://github.com/cdapio/cdap/blob/develop/cdap-app-fabric/src/main/java/co/cask/cdap/gateway/handlers/TransactionHttpHandler.java#L97 A tool like that would be a great contribution for Tephra. Cheers -Andreas On Tue, Jan 22, 2019 at 3:44 PM Varun Rao wrote: > Hello, > > We are interested in reading snapshot files from an HDFS directory and we > want to know if this was already provided and if not, is it even possible? > Within the HDFSTransactionStateStorage.java file , the main method looks as > if it supports a CLI tool that reads a transaction state snapshot or > transaction log from HDFS. I have left out extraneous code but the for loop > below iterates through a list of files (each file pointing to a transaction > state snapshot or transaction log) and prints each snapshot to stdout. > > 1) Is this CLI tool existent, or on the roadmap to be developed? > 2) Is it safe to call this main method to read snapshots from an HDFS > directory? > > // TODO move this out as a separate command line tool > private enum CLIMode { SNAPSHOT, TXLOG }; > > /** > * Reads a transaction state snapshot or transaction log from HDFS and > prints the entries to stdout. > * > * Supports the following options: > * -s read snapshot state (defaults to the latest) > * -l read a transaction log > * [filename] reads the given file > * @param args > */ > > public static void main(String[] args) { > > *//......... * > > for (String file : filenames) { > Path path = new Path(file); > TransactionSnapshot snapshot = storage.readSnapshotFile(path); > printSnapshot(snapshot); > > } > > } > > Thanks very much > --0000000000008edcc90580169811--