From commits-return-100983-apmail-lucene-commits-archive=www.apache.org@lucene.apache.org Tue May 15 20:04:22 2018 Return-Path: X-Original-To: apmail-lucene-commits-archive@www.apache.org Delivered-To: apmail-lucene-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DAF0718AD6 for ; Tue, 15 May 2018 20:04:22 +0000 (UTC) Received: (qmail 23495 invoked by uid 500); 15 May 2018 20:04:19 -0000 Mailing-List: contact commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list commits@lucene.apache.org Received: (qmail 22827 invoked by uid 99); 15 May 2018 20:04:19 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 May 2018 20:04:18 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 8B8D2F6D4A; Tue, 15 May 2018 20:04:18 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: ab@apache.org To: commits@lucene.apache.org Date: Tue, 15 May 2018 20:04:33 -0000 Message-Id: <5c15f7df6d1f485182219bd862a66a4e@git.apache.org> In-Reply-To: <285d72d4f32d4fbd94143810b0c72123@git.apache.org> References: <285d72d4f32d4fbd94143810b0c72123@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [16/39] lucene-solr:jira/solr-11779: Squashed commit of the following: Squashed commit of the following: commit e5074c3223e394af17f686294a67a1dd3ecdd147 Author: Joel Bernstein Date: Wed May 9 13:16:34 2018 -0400 SOLR-12280: WIP commit 69cdeccf161177d10f4d2407542392aaee3fcfe8 Author: Joel Bernstein Date: Wed May 9 13:08:02 2018 -0400 SOLR-12280: WIP commit c94f0c87c3e57c023d622ad2411e522c4aac491c Author: Joel Bernstein Date: Wed May 9 11:54:58 2018 -0400 SOLR-12280: WIP commit 68dd1e73355cb84410f2d0ff3a51797ed6194a10 Author: Joel Bernstein Date: Wed May 9 10:54:32 2018 -0400 SOLR-12280: WIP commit 04a010543418a469100fa299c606a7b1eed452e1 Author: Joel Bernstein Date: Wed May 9 10:47:27 2018 -0400 SOLR-12280: WIP commit a6bbfbadaafe33fcdf93d5c72755e30dadadf017 Author: Joel Bernstein Date: Wed May 9 09:40:08 2018 -0400 SOLR-12280: WIP commit 5d27961aa291bcd71527337632981bcdf62369b4 Author: Joel Bernstein Date: Tue May 8 20:43:33 2018 -0400 SOLR-12280: WIP commit e982cf939f429c05b736f6292c68dd96d7ebc027 Author: Joel Bernstein Date: Tue May 8 13:27:29 2018 -0400 SOLR-12280: WIP commit aae78ab6f387c28a080021bc919ef51864540be2 Author: Joel Bernstein Date: Tue May 8 12:23:52 2018 -0400 SOLR-12280: WIP commit 0787ad76f0f4c62c860784b15490d8a988939997 Author: Joel Bernstein Date: Tue May 8 12:20:38 2018 -0400 SOLR-12280: WIP commit 4df098376ba05188702cca8582959c3fe18066f5 Author: Joel Bernstein Date: Tue May 8 12:12:11 2018 -0400 SOLR-12280: WIP commit 5c0be5136bbab7e0c33b3b8a7b0395b1b330e96d Author: Joel Bernstein Date: Tue May 8 12:04:57 2018 -0400 SOLR-12280: WIP commit 6c6feac4c2e5a49a5eab87a228713d1b93c8fc70 Author: Joel Bernstein Date: Tue May 8 11:57:49 2018 -0400 SOLR-12280: WIP commit 7d46d11c9dd3a51b68600c2c889f586147545294 Author: Joel Bernstein Date: Tue May 8 11:50:51 2018 -0400 SOLR-12280: WIP commit 8b6bf19d0091203ed63b39d070dd02a9bece6a61 Author: Joel Bernstein Date: Mon May 7 10:53:14 2018 -0400 SOLR-12280: WIP commit 5466591999816eaacde6ce18d824d7688e5f4fe8 Author: Joel Bernstein Date: Fri May 4 15:12:43 2018 -0400 SOLR-12280: WIP commit d7fff7d557a7fd26011c21445b7969b2cd81036f Author: Joel Bernstein Date: Fri Apr 27 12:50:27 2018 -0400 SOLR-12280: Initial commit Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/144f00a1 Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/144f00a1 Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/144f00a1 Branch: refs/heads/jira/solr-11779 Commit: 144f00a1e315541a28f526f4cdf1e55eb60c862b Parents: 4177252 Author: Joel Bernstein Authored: Wed May 9 13:24:08 2018 -0400 Committer: Joel Bernstein Committed: Wed May 9 13:24:08 2018 -0400 ---------------------------------------------------------------------- solr/solr-ref-guide/src/dsp.adoc | 719 +++++++++++++++++++ .../hidden-signal-autocorrelation.png | Bin 0 -> 258831 bytes .../math-expressions/hidden-signal-fft.png | Bin 0 -> 215981 bytes .../images/math-expressions/hidden-signal.png | Bin 0 -> 319100 bytes .../math-expressions/noise-autocorrelation.png | Bin 0 -> 204511 bytes .../src/images/math-expressions/noise-fft.png | Bin 0 -> 319551 bytes .../src/images/math-expressions/noise.png | Bin 0 -> 375565 bytes .../math-expressions/signal-autocorrelation.png | Bin 0 -> 322164 bytes .../src/images/math-expressions/signal-fft.png | Bin 0 -> 140111 bytes .../src/images/math-expressions/signal.png | Bin 0 -> 365018 bytes solr/solr-ref-guide/src/math-expressions.adoc | 2 + 11 files changed, 721 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/dsp.adoc ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/dsp.adoc b/solr/solr-ref-guide/src/dsp.adoc new file mode 100644 index 0000000..348030e --- /dev/null +++ b/solr/solr-ref-guide/src/dsp.adoc @@ -0,0 +1,719 @@ += Digital Signal Processing +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +This section of the user guide explores functions that are commonly used in the field of +Digital Signal Processing (DSP). + +== Dot product + +The `dotProduct` function is used to calculate the dot product of two arrays. +The dot product is a fundamental calculation for the DSP functions discussed in this section. Before diving into +the more advanced DSP functions, its useful to get a better understanding of how the dot product calculation works. + +=== Combining two arrays + +The `dotProduct` function can be used to combine two arrays into a single product. A simple example can help +illustrate this concept. + +In the example below two arrays are set to variables *a* and *b* and then operated on by the `dotProduct` function. +The output of the `dotProduct` function is set to variable *c*. + +Then the `mean` function is then used to compute the mean of the first array which is set to the variable `d`. + +Both the *dot product* and the *mean* are included in the output. + +When we look at the output of this expression we see that the *dot product* and the *mean* of the first array +are both 30. + +The dot product function *calculated the mean* of the first array. + +[source,text] +---- +let(echo="c, d", + a=array(10, 20, 30, 40, 50), + b=array(.2, .2, .2, .2, .2), + c=dotProduct(a, b), + d=mean(a)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "c": 30, + "d": 30 + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +To get a better understanding of how the dot product calculated the mean we can perform the steps of the +calculation using vector math and look at the output of each step. + +In the example below the `ebeMultiply` function performs an element-by-element multiplication of +two arrays. This is the first step of the dot product calculation. The result of the element-by-element +multiplication is assigned to variable *c*. + +In the next step the `add` function adds all the elements of the array in variable *c*. + +Notice that multiplying each element of the first array by .2 and then adding the results is +equivalent to the formula for computing the mean of the first array. The formula for computing the mean +of an array is to add all the elements and divide by the number of elements. + +The output includes the output of both the `ebeMultiply` function and the `add` function. + +[source,text] +---- +let(echo="c, d", + a=array(10, 20, 30, 40, 50), + b=array(.2, .2, .2, .2, .2), + c=ebeMultiply(a, b), + d=add(c)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "c": [ + 2, + 4, + 6, + 8, + 10 + ], + "d": 30 + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +In the example above two arrays were combined in a way that produced the mean of the first. In the second array +each value was set to .2. Another way of looking at this is that each value in the second array has the same weight. +By varying the weights in the second array we can produce a different result. For example if the first array represents a time series, +the weights in the second array can be set to add more weight to a particular element in the first array. + +The example below creates a weighted average with the weight decreasing from right to left. Notice that the weighted mean +of 36.666 is larger than the previous mean which was 30. This is because more weight was given to last element in the +array. + +[source,text] +---- +let(echo="c, d", + a=array(10, 20, 30, 40, 50), + b=array(.066666666666666,.133333333333333,.2, .266666666666666, .33333333333333), + c=ebeMultiply(a, b), + d=add(c)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "c": [ + 0.66666666666666, + 2.66666666666666, + 6, + 10.66666666666664, + 16.6666666666665 + ], + "d": 36.66666666666646 + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +=== Representing Correlation + +Often when we think of correlation, we are thinking of *Pearsons* correlation in the field of statistics. But the definition of +correlation is actually more general: a mutual relationship or connection between two or more things. +In the field of digital signal processing the dot product is used to represent correlation. The examples below demonstrates +how the dot product can be used to represent correlation. + +In the example below the dot product is computed for two vectors. Notice that the vectors have different values that fluctuate +together. The output of the dot product is 190, which is hard to reason about because because its not scaled. + +[source,text] +---- +let(echo="c, d", + a=array(10, 20, 30, 20, 10), + b=array(1, 2, 3, 2, 1), + c=dotProduct(a, b)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "c": 190 + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +One approach to scaling the dot product is to first scale the vectors so that both vectors have a magnitude of 1. Vectors with a +magnitude of 1, also called unit vectors, are used when comparing only the angle between vectors rather then the magnitude. +The `unitize` function can be used to unitize the vectors before calculating the dot product. + +Notice in the example below the dot product result, set to variable *e*, is effectively 1. When applied to unit vectors the dot product +will be scaled between 1 and -1. Also notice in the example `cosineSimilarity` is calculated on the *unscaled* vectors and the +answer is also effectively 1. This is because *cosine similarity* is a scaled *dot product*. + + +[source,text] +---- +let(echo="e, f", + a=array(10, 20, 30, 20, 10), + b=array(1, 2, 3, 2, 1), + c=unitize(a), + d=unitize(b), + e=dotProduct(c, d), + f=cosineSimilarity(a, b)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "e": 0.9999999999999998, + "f": 0.9999999999999999 + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +If we transpose the first two numbers in the first array, so that the vectors +are not perfectly correlated, we see that the cosine similarity drops. This illustrates +how the dot product represents correlation. + +[source,text] +---- +let(echo="c, d", + a=array(20, 10, 30, 20, 10), + b=array(1, 2, 3, 2, 1), + c=cosineSimilarity(a, b)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "c": 0.9473684210526314 + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +== Convolution + +The `conv` function calculates the convolution of two vectors. The convolution is calculated by *reversing* +the second vector and sliding it across the first vector. The *dot product* of the two vectors +is calculated at each point as the second vector is slid across the first vector. +The dot products are collected in a *third vector* which is the *convolution* of the two vectors. + +=== Moving Average + +Before looking at an example of convolution its useful to review the `movingAvg` function. The moving average +function computes a moving average by sliding a window across a vector and computing +the average of the window at each shift. If that sounds similar to convolution, that's because the `movingAvg` function +is syntactic sugar for convolution. + +Below is an example of a moving average with a window size of 5. Notice that original vector has 13 elements +but the result of the moving average has only 9 elements. This is because the `movingAvg` function +only begins generating results when it has a full window. In this case because the window size is 5 so the +moving average starts generating results from the 4th index of the original array. + +[source,text] +---- +let(a=array(1, 2, 3, 4, 5, 6, 7, 6, 5, 4, 3, 2, 1), + b=movingAvg(a, 5)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "b": [ + 3, + 4, + 5, + 5.6, + 5.8, + 5.6, + 5, + 4, + 3 + ] + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +=== Convolutional Smoothing + +The moving average can also be computed using convolution. In the example +below the `conv` function is used to compute the moving average of the first array +by applying the second array as the filter. + +Looking at the result, we see that it is not exactly the same as the result +of the `movingAvg` function. That is because the `conv` pads zeros +to the front and back of the first vector so that the window size is always full. + +[source,text] +---- +let(a=array(1, 2, 3, 4, 5, 6, 7, 6, 5, 4, 3, 2, 1), + b=array(.2, .2, .2, .2, .2), + c=conv(a, b)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "c": [ + 0.2, + 0.6000000000000001, + 1.2, + 2.0000000000000004, + 3.0000000000000004, + 4, + 5, + 5.6000000000000005, + 5.800000000000001, + 5.6000000000000005, + 5.000000000000001, + 4, + 3, + 2, + 1.2000000000000002, + 0.6000000000000001, + 0.2 + ] + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +We achieve the same result as the `movingAvg` gunction by using the `copyOfRange` function to copy a range of +the result that drops the first and last 4 values of +the convolution result. In the example below the `precision` function is also also used to remove floating point errors from the +convolution result. When this is added the output is exactly the same as the `movingAvg` function. + +[source,text] +---- +let(a=array(1, 2, 3, 4, 5, 6, 7, 6, 5, 4, 3, 2, 1), + b=array(.2, .2, .2, .2, .2), + c=conv(a, b), + d=copyOfRange(c, 4, 13), + e=precision(d, 2)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "e": [ + 3, + 4, + 5, + 5.6, + 5.8, + 5.6, + 5, + 4, + 3 + ] + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +== Cross-Correlation + +Cross-correlation is used to determine the delay between two signals. This is accomplished by sliding one signal across another +and calculating the dot product at each shift. The dot products are collected into a vector which represents the correlation +at each shift. The highest dot product in the cross-correlation vector is the point where the two signals are most closely correlated. + +The sliding dot product used in convolution can also be used to represent cross-correlation between two vectors. The only +difference in the formula when representing correlation is that the second vector is *not reversed*. + +Notice in the example below that the second vector is reversed by the `rev` function before it is operated on by the `conv` function. +The `conv` function reverses the second vector so it will be flipped back to its original order to perform the correlation calculation +rather then the convolution calculation. + +Notice in the result the highest value is 217. This is the point where the two vectors have the highest correlation. + +[source,text] +---- +let(a=array(1, 2, 3, 4, 5, 6, 7, 6, 5, 4, 3, 2, 1), + b=array(4, 5, 6, 7, 6, 5, 4, 3, 2, 1), + c=conv(a, rev(b))) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "c": [ + 1, + 4, + 10, + 20, + 35, + 56, + 84, + 116, + 149, + 180, + 203, + 216, + 217, + 204, + 180, + 148, + 111, + 78, + 50, + 28, + 13, + 4 + ] + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +== Find Delay + +It is fairly simple to compute the delay from the cross-correlation result, but a convenience function called `finddelay` can +be used to find the delay directly. Under the covers `finddelay` uses convolutional math to compute the cross-correlation vector +and then computes the delay between the two signals. + +Below is an example of the `finddelay` function. Notice that the `finddelay` function reports a 3 period delay between the first +and second signal. + +[source,text] +---- +let(a=array(1, 2, 3, 4, 5, 6, 7, 6, 5, 4, 3, 2, 1), + b=array(4, 5, 6, 7, 6, 5, 4, 3, 2, 1), + c=finddelay(a, b)) +---- + +When this expression is sent to the /stream handler it responds with: + +[source,json] +---- +{ + "result-set": { + "docs": [ + { + "c": 3 + }, + { + "EOF": true, + "RESPONSE_TIME": 0 + } + ] + } +} +---- + +== Autocorrelation + +Autocorrelation measures the degree to which a signal is correlated with itself. Autocorrelation is used to determine +if a vector contains a signal or is purely random. + +A few examples, with plots, will help to understand the concepts. + +In the first example the `sin` function is wrapped around a `sequence` function to generate a sine wave. The result of this +is plotted in the image below. Notice that there is a structure to the plot that is clearly not random. + +[source,text] +---- +sin(sequence(256, 0, 6)) +---- + +image::images/math-expressions/signal.png[] + + +In the next example the `sample` function is used to draw 256 samples from a `uniformDistribution` to create a +vector of random data. The result of this is plotted in the image below. Notice that there is no clear structure to the +data and the data appears to be random. + +[source,text] +---- +sample(uniformDistribution(-1.5, 1.5), 256) +---- + +image::images/math-expressions/noise.png[] + + +In the next example the random noise is added to the sine wave using the `ebeAdd` function. +The result of this is plotted in the image below. Notice that the sine wave has been hidden +somewhat within the noise. Its difficult to say for sure if there is structure. As plots +becomes more dense it can become harder to see a pattern hidden within noise. + +[source,text] +---- +let(a=sin(sequence(256, 0, 6)), + b=sample(uniformDistribution(-1.5, 1.5), 256), + c=ebeAdd(a, b)) +---- + +image::images/math-expressions/hidden-signal.png[] + + +In the next examples autocorrelation is performed with each of the vectors shown above to see what the +autocorrelation plots look like. + +In the example below the `conv` function is used to autocorrelate the first vector which is the sine wave. +Notice that the `conv` function is simply correlating the sine wave with itself. + +The plot has a very distinct structure to it. As the sine wave is slid across a copy of itself the correlation +moves up and down in increasing intensity until it reaches a peak. This peak is directly in the center and is the +the point where the sine waves are directly lined up. Following the peak the correlation moves up and down in decreasing +intensity as the sine wave slides farther away from being directly lined up. + +This is the autocorrelation plot of a pure signal. + +[source,text] +---- +let(a=sin(sequence(256, 0, 6)), + b=conv(a, rev(a)), +---- + +image::images/math-expressions/signal-autocorrelation.png[] + + +In the example below autocorrelation is performed with the vector of pure noise. Notice that the autocorrelation +plot has a very different plot then the sine wave. In this plot there is long period of low intensity correlation that appears +to be random. Then in the center a peak of high intensity correlation where the vectors are directly lined up. +This is followed by another long period of low intensity correlation. + +This is the autocorrelation plot of pure noise. + +[source,text] +---- +let(a=sample(uniformDistribution(-1.5, 1.5), 256), + b=conv(a, rev(a)), +---- + +image::images/math-expressions/noise-autocorrelation.png[] + + +In the example below autocorrelation is performed on the vector with the sine wave hidden within the noise. +Notice that this plot shows very clear signs of structure which is similar to autocorrelation plot of the +pure signal. The correlation is less intense due to noise but the shape of the correlation plot suggests +strongly that there is an underlying signal hidden within the noise. + +[source,text] +---- +let(a=sin(sequence(256, 0, 6)), + b=sample(uniformDistribution(-1.5, 1.5), 256), + c=ebeAdd(a, b), + d=conv(c, rev(c)) +---- + +image::images/math-expressions/hidden-signal-autocorrelation.png[] + + +== Discrete Fourier Transform + +The convolution based functions described above are operating on signals in the time domain. In the time +domain the X axis is time and the Y axis is the quantity of some value at a specific point in time. + +The discrete Fourier Transform translates a time domain signal into the frequency domain. +In the frequency domain the X axis is frequency, and Y axis is the accumulated power at a specific frequency. + +The basic principle is that every time domain signal is composed of one or more signals (sine waves) +at different frequencies. The discrete Fourier transform decomposes a time domain signal into its component +frequencies and measures the power at each frequency. + +The discrete Fourier transform has many important uses. In the example below, the discrete Fourier transform is used +to determine if a signal has structure or if it is purely random. + +=== Complex Result + +The `fft` function performs the discrete Fourier Transform on a vector of *real* data. The result +of the `fft` function is returned as *complex* numbers. A complex number has two parts, *real* and *imaginary*. +The imaginary part of the complex number is ignored in the examples below, but there +are many tutorials on the FFT and that include complex numbers available online. + +But before diving into the examples it is important to understand how the `fft` function formats the +complex numbers in the result. + +The `fft` function returns a `matrix` with two rows. The first row in the matrix is the *real* +part of the complex result. The second row in the matrix is the *imaginary* part of the complex result. + +The `rowAt` function can be used to access the rows so they can be processed as vectors. +This approach was taken because all of the vector math functions operate on vectors of real numbers. +Rather then introducing a complex number abstraction into the expression language, the `fft` result is +represented as two vectors of real numbers. + +=== Fast Fourier Transform Examples + +In the first example the `fft` function is called on the sine wave used in the autocorrelation example. + +The results of the `fft` function is a matrix. The `rowAt` function is used to return the first row of +the matrix which is a vector containing the real values of the fft response. + +The plot of the real values of the `fft` response is shown below. Notice there are two +peaks on opposite sides of the plot. The plot is actually showing a mirrored response. The right side +of the plot is an exact mirror of the left side. This is expected when the `fft` is run on real rather then +complex data. + +Also notice that the `fft` has accumulated significant power in a single peak. This is the power associated with +the specific frequency of the sine wave. The vast majority of frequencies in the plot have close to 0 power +associated with them. This `fft` shows a clear signal with very low levels of noise. + +[source,text] +---- +let(a=sin(sequence(256, 0, 6)), + b=fft(a), + c=rowAt(b, 0)) +---- + + +image::images/math-expressions/signal-fft.png[] + +In the second example the `fft` function is called on a vector of random data similar to one used in the +autocorrelation example. The plot of the real values of the `fft` response is shown below. + +Notice that in is this response there is no clear peak. Instead all frequencies have accumulated a random level of +power. This `fft` shows no clear sign of signal and appears to be noise. + + +[source,text] +---- +let(a=sample(uniformDistribution(-1.5, 1.5), 256), + b=fft(a), + c=rowAt(b, 0)) +---- + +image::images/math-expressions/noise-fft.png[] + + +In the third example the `fft` function is called on the same signal hidden within noise that was used for +the autocorrelation example. The plot of the real values of the `fft` response is shown below. + +Notice that there are two clear mirrored peaks, at the same locations as the `fft` of the pure signal. But +there is also now considerable noise on the frequencies. The `fft` has found the signal and but also +shows that there is considerable noise along with the signal. + +[source,text] +---- +let(a=sin(sequence(256, 0, 6)), + b=sample(uniformDistribution(-1.5, 1.5), 256), + c=ebeAdd(a, b), + d=fft(c), + e=rowAt(d, 0)) +---- + +image::images/math-expressions/hidden-signal-fft.png[] + + http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/images/math-expressions/hidden-signal-autocorrelation.png ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/images/math-expressions/hidden-signal-autocorrelation.png b/solr/solr-ref-guide/src/images/math-expressions/hidden-signal-autocorrelation.png new file mode 100644 index 0000000..f741c18 Binary files /dev/null and b/solr/solr-ref-guide/src/images/math-expressions/hidden-signal-autocorrelation.png differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/images/math-expressions/hidden-signal-fft.png ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/images/math-expressions/hidden-signal-fft.png b/solr/solr-ref-guide/src/images/math-expressions/hidden-signal-fft.png new file mode 100644 index 0000000..58b0c60 Binary files /dev/null and b/solr/solr-ref-guide/src/images/math-expressions/hidden-signal-fft.png differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/images/math-expressions/hidden-signal.png ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/images/math-expressions/hidden-signal.png b/solr/solr-ref-guide/src/images/math-expressions/hidden-signal.png new file mode 100644 index 0000000..9baff48 Binary files /dev/null and b/solr/solr-ref-guide/src/images/math-expressions/hidden-signal.png differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/images/math-expressions/noise-autocorrelation.png ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/images/math-expressions/noise-autocorrelation.png b/solr/solr-ref-guide/src/images/math-expressions/noise-autocorrelation.png new file mode 100644 index 0000000..d69a9a2 Binary files /dev/null and b/solr/solr-ref-guide/src/images/math-expressions/noise-autocorrelation.png differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/images/math-expressions/noise-fft.png ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/images/math-expressions/noise-fft.png b/solr/solr-ref-guide/src/images/math-expressions/noise-fft.png new file mode 100644 index 0000000..cdcfba3 Binary files /dev/null and b/solr/solr-ref-guide/src/images/math-expressions/noise-fft.png differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/images/math-expressions/noise.png ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/images/math-expressions/noise.png b/solr/solr-ref-guide/src/images/math-expressions/noise.png new file mode 100644 index 0000000..6b4f976 Binary files /dev/null and b/solr/solr-ref-guide/src/images/math-expressions/noise.png differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/images/math-expressions/signal-autocorrelation.png ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/images/math-expressions/signal-autocorrelation.png b/solr/solr-ref-guide/src/images/math-expressions/signal-autocorrelation.png new file mode 100644 index 0000000..cd24667 Binary files /dev/null and b/solr/solr-ref-guide/src/images/math-expressions/signal-autocorrelation.png differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/images/math-expressions/signal-fft.png ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/images/math-expressions/signal-fft.png b/solr/solr-ref-guide/src/images/math-expressions/signal-fft.png new file mode 100644 index 0000000..f70fa46 Binary files /dev/null and b/solr/solr-ref-guide/src/images/math-expressions/signal-fft.png differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/images/math-expressions/signal.png ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/images/math-expressions/signal.png b/solr/solr-ref-guide/src/images/math-expressions/signal.png new file mode 100644 index 0000000..cf2b933 Binary files /dev/null and b/solr/solr-ref-guide/src/images/math-expressions/signal.png differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/144f00a1/solr/solr-ref-guide/src/math-expressions.adoc ---------------------------------------------------------------------- diff --git a/solr/solr-ref-guide/src/math-expressions.adoc b/solr/solr-ref-guide/src/math-expressions.adoc index 3f9ed1a..0b8fafc 100644 --- a/solr/solr-ref-guide/src/math-expressions.adoc +++ b/solr/solr-ref-guide/src/math-expressions.adoc @@ -56,4 +56,6 @@ record in your Solr Cloud cluster computable. == <> +== <> + == <>