From dev-return-367351-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Fri Oct 18 20:43:04 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 612321804BB for ; Fri, 18 Oct 2019 22:43:04 +0200 (CEST) Received: (qmail 70508 invoked by uid 500); 18 Oct 2019 20:43:03 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 70496 invoked by uid 99); 18 Oct 2019 20:43:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Oct 2019 20:43:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 25F1FC1CFA for ; Fri, 18 Oct 2019 20:43:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.001 X-Spam-Level: * X-Spam-Status: No, score=1.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, KAM_LIVE=1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id QBI9T36En8pG for ; Fri, 18 Oct 2019 20:43:00 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.48; helo=mail-lf1-f48.google.com; envelope-from=joelsolr@gmail.com; receiver= Received: from mail-lf1-f48.google.com (mail-lf1-f48.google.com [209.85.167.48]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id BBDA3BC5B1 for ; Fri, 18 Oct 2019 20:42:59 +0000 (UTC) Received: by mail-lf1-f48.google.com with SMTP id t8so5634351lfc.13 for ; Fri, 18 Oct 2019 13:42:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=0wm8K7q/etiS94vaIL/8Jov/90I20UQ68aJspplqpvg=; b=dUmELk7YkxrWtGBCn4909bjkLE0gJuzb8H2SHmWZ4SLmsycXdKaAu8xYE7mgBKLily ZVpseqHonn2Bo1/yo7QxItmrOKYgdK6S/KloMjqoIoFzQecgz22Y0y3EBAWvWrp7Y0zb lkPQF9pKQfhmfflmkd81wVvunPeDa6WrJeEQNPzPfEudcaK8La9kQ76UZKFnsV/UUjiv mKyVVBmAtpfFogOBgZwbIqslKunCnGSt8YO137fXkv3r4Vqcr0zFOmrYSvlCbxJM4cK3 FwBxFDpJerTRxW36wzOiocZknY9toZl0CMKBOm/H2P69TvpdyJzEF/5bPMxQ+4Ikae9Q ZbNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=0wm8K7q/etiS94vaIL/8Jov/90I20UQ68aJspplqpvg=; b=PjCZXm9VDlvDI4FHhHTJBgrP34H3IU3Zhk3B4vkvevegknm+e+f6syYreNj+XrAIRe KK69kDY6ZRLFdmNIK6AU0VRbSKETPZRp1MiU7pQ/RmO2Qc5nZ3FoLdmIi6+mrcL3E57v yb46kex3aKiCkA+SQrjE5aHeZ10Z4ghsUbVLOFsSg+/G8bWMoF1tvTjm1V/ime0chYhq wrTs8ni0qg/zqxXj4MPtU61uFysEIkR7V9UdXkuTvzDfBolzkJlBp3nbsOdk9YvVGo1k IGGQdYqME9+hwbyS4tJNecgIKx2nC7OdkZ0BI06U8UyUnE5sxy84sESzL4/peSBrjyet l4yg== X-Gm-Message-State: APjAAAXUfLTfaixZe96Lq1+6a0leFmHBM8bx5cho9tqHPXfej03HZKel IEjiP1BqhptTUH/Jts0FZeyC4aDAA1DERa4HgUawoCgS X-Google-Smtp-Source: APXvYqxB4wH9VCHK/g8GvrNT9nP1pYVzQtCmtu5oqK5QNmG7KJ34ARak2mG9X9fICBSrJIqNSnw6xGCbw8bbUc2/c+A= X-Received: by 2002:a19:23d7:: with SMTP id j206mr5279733lfj.187.1571431378097; Fri, 18 Oct 2019 13:42:58 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Joel Bernstein Date: Fri, 18 Oct 2019 16:42:46 -0400 Message-ID: Subject: Re: Capturing URL params for use within Streaming Expressions To: lucene dev Content-Type: multipart/alternative; boundary="000000000000149b9d059535611b" --000000000000149b9d059535611b Content-Type: text/plain; charset="UTF-8" I think it's fine to pass through parameters to the various stream sources. Perhaps we should limit it to a set list of parameters to pass through just so it limits the scope, Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Oct 16, 2019 at 4:47 PM Houston Putman wrote: > Streaming expressions allow for users to pass in any arbitrary URL params > in the search streaming source. I'm looking to add the ability for certain > streaming functions, maybe just "search()" but possibly more, to extract > the extra URL params passed along in the streaming request. > > For example sending a request: > http://localhost:8983/solr/example/stream?expr=search(collection1, > q="*:*", fl="id", sort="id")& > *shards.preference=shards.preference=replica.location:local* > > would be equivalent to: > http://localhost:8983/solr/example/stream?expr=search(collection1, > q="*:*", fl="id", sort="id", > *shards.preference="shards.preference=replica.location:local"*) > > The beauty of URL params is that they can easily be overridden and > checked, for example in a proxy between the user and solr. It is harder to > do this with streaming expressions as the proxy would need to parse the > expression and know the logic of the functions and sources. > > I'm open to discussion on whether the params able to be captured by the > streaming function would need to be white-listed or black-listed. My idea > is that this would be generically implemented through something like the > StreamContext, so that any streaming function that wants to add this > functionality is able to do so. > > Another option is to add a URL parameter such as > *&expr.override.search.shards.preference=replica.location:local* ( > *expr.override..=*). That way it's explicit > that the user is trying to send options to the streaming expression, and > extraneous URL params aren't accidentally captured when they were included > for a different purpose. > > Anyways this would really help us for some uses cases, especially the > replica routing options used in the example above. Really interested to see > opinions on either of these options. > > - Houston Putman > --000000000000149b9d059535611b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I think it's fine to pass through parameters to t= he various stream sources. Perhaps we should limit it to a set list of para= meters to pass through just so it limits the scope,


<= br>
On Wed,= Oct 16, 2019 at 4:47 PM Houston Putman <houstonputman@gmail.com> wrote:
Streaming=C2=A0expressi= ons allow for users to pass in any arbitrary URL params in the search strea= ming source. I'm looking to add the ability for certain streaming funct= ions, maybe just "search()" but possibly more, to extract the ext= ra URL params passed along in the streaming request.

For example sen= ding a request:
http://localhost:8983/solr/e= xample/stream?expr=3Dsearch(collection1, q=3D"*:*", fl=3D&quo= t;id", sort=3D"id")&shards.preference=3Dshards.prefer= ence=3Dreplica.location:local

would be equivalent to= :
http://localhost:8983/solr/example= /stream?expr=3Dsearch(collection1, q=3D"*:*", fl=3D"id&q= uot;, sort=3D"id", shards.preference=3D"shards.preference= =3Dreplica.location:local")

The beauty of URL params i= s that they can easily be overridden and checked, for=C2=A0example in a pro= xy between the user and solr. It is harder to do this with streaming expres= sions as the proxy would need to parse the expression and know the logic of= the functions and sources.

I'm open to discussion on whether th= e params able to be captured by the streaming function would need to be whi= te-listed or black-listed. My idea is that this would be generically implem= ented through something like the StreamContext, so that any streaming funct= ion that wants to add this functionality is able to do so.

Another o= ption is to add a URL parameter such as &expr.override.search.shards= .preference=3Dreplica.location:local=C2=A0(expr.override.<functio= n>.<parameter>=3D<value>). That way it's explicit th= at the user is trying to send options to the streaming expression, and extr= aneous URL params aren't accidentally captured when they were included = for a different purpose.

Anyways this would really help us for some = uses cases, especially the replica routing options used in the example abov= e. Really interested to see opinions on either of these options.

- Houston Putman
--000000000000149b9d059535611b--