Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id F302A200C6F for ; Mon, 24 Apr 2017 19:37:08 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id F1C27160B93; Mon, 24 Apr 2017 17:37:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4CFF3160B99 for ; Mon, 24 Apr 2017 19:37:08 +0200 (CEST) Received: (qmail 27841 invoked by uid 500); 24 Apr 2017 17:37:07 -0000 Mailing-List: contact dev-help@apex.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@apex.apache.org Delivered-To: mailing list dev@apex.apache.org Received: (qmail 27772 invoked by uid 99); 24 Apr 2017 17:37:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Apr 2017 17:37:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 23E5E1AFFCF for ; Mon, 24 Apr 2017 17:37:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id S5tF4NFtnsxW for ; Mon, 24 Apr 2017 17:37:05 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id E7A265FC7E for ; Mon, 24 Apr 2017 17:37:04 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 7CC38E095D for ; Mon, 24 Apr 2017 17:37:04 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2D40421B56 for ; Mon, 24 Apr 2017 17:37:04 +0000 (UTC) Date: Mon, 24 Apr 2017 17:37:04 +0000 (UTC) From: "Chaitanya (JIRA)" To: dev@apex.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (APEXMALHAR-2484) BlockWriter for writing the part files into the specified directory MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 24 Apr 2017 17:37:09 -0000 Chaitanya created APEXMALHAR-2484: ------------------------------------- Summary: BlockWriter for writing the part files into the specified directory Key: APEXMALHAR-2484 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2484 Project: Apache Apex Malhar Issue Type: Task Reporter: Chaitanya Assignee: Chaitanya Use case: Suppose, the size of source file (f1.txt) is 1 GB and the block size is 128 MB. I want to copy the file in destination as follows: f1.txt.part1 f2.txt.part2 .... By default, size of each part file is 128 MB except the last part. Design: Currently, the BlockWriter is restricted to write the part files into the HDFS on which the app is running. To achieve the above use case, operator needs the block index and relative path information. BlockMetadata which is the input port for the BlockWriter doesn't have these information. So, I am creating the new operator(PartFileWriter) which extends from BlockWriter with the input port of type FileMetadata. -- This message was sent by Atlassian JIRA (v6.3.15#6346)