From issues-return-194834-archive-asf-public=cust-asf.ponee.io@spark.apache.org Tue Jun 26 00:10:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2F09718066B for ; Tue, 26 Jun 2018 00:10:04 +0200 (CEST) Received: (qmail 53408 invoked by uid 500); 25 Jun 2018 22:10:03 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 53322 invoked by uid 99); 25 Jun 2018 22:10:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jun 2018 22:10:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D78AFC0264 for ; Mon, 25 Jun 2018 22:10:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id vJyt-TlJ1hUW for ; Mon, 25 Jun 2018 22:10:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id B305A5FAC2 for ; Mon, 25 Jun 2018 22:10:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 2263EE0FDB for ; Mon, 25 Jun 2018 22:10:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 6052423FA3 for ; Mon, 25 Jun 2018 22:10:00 +0000 (UTC) Date: Mon, 25 Jun 2018 22:10:00 +0000 (UTC) From: "Michael Dreibelbis (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (SPARK-24656) SparkML Transformers and Estimators with multiple columns MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Michael Dreibelbis created SPARK-24656: ------------------------------------------ Summary: SparkML Transformers and Estimators with multiple col= umns Key: SPARK-24656 URL: https://issues.apache.org/jira/browse/SPARK-24656 Project: Spark Issue Type: New Feature Components: ML, MLlib Affects Versions: 2.3.1 Reporter: Michael Dreibelbis Currently SparkML Transformers and Estimators operate on single input/outpu= t column pairs. This makes pipelines extremely cumbersome (as well as non-p= erformant) when transformations on multiple columns needs to be made. =C2=A0 I am proposing to implement ParallelPipelineStage/Transformer/Estimator/Mod= el that would operate on the input columns in parallel. =C2=A0 {code:java} // old way val pipeline =3D new Pipeline().setStages(Array( new CountVectorizer().setInputCol("_1").setOutputCol("_1_cv"), new CountVectorizer().setInputCol("_2").setOutputCol("_2_cv"), new IDF().setInputCol("_1_cv").setOutputCol("_1_idf"), new IDF().setInputCol("_2_cv").setOutputCol("_2_idf") )) // proposed way val pipeline2 =3D new Pipeline().setStages(Array( new ParallelCountVectorizer().setInputCols(Array("_1", "_2")).setOutp= utCols(Array("_1_cv", "_2_cv")), new ParallelIDF().setInputCols(Array("_1_cv", "_2_cv")).setOutputCols= (Array("_1_idf", "_2_idf")) )) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org