predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shane Johnson <shanewaldenjohn...@gmail.com>
Subject Re: Adding Spark Context (sc) to predict method?
Date Thu, 05 Oct 2017 21:23:33 GMT
Thanks Daniel. I may be missing what you are saying. I actually think I
need the Spark Context for what I am trying to do. I am wanting to extend
the predict method to use id's from the Query object and then go back into
the EventStore to get additional attributes that were not passed in the
Query.

To do this I need to have a sc or I cannot get the properties from the
EventStore. Does that make sense. If I remove the Spark Context it does
indeed work but not for what I am trying to do.

Thanks

*Shane Johnson | 801.360.3350*
LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook
<https://www.facebook.com/shane.johnson.71653>

On Thu, Oct 5, 2017 at 3:19 PM, Daniel O' Shaughnessy <
danieljamesdavid@gmail.com> wrote:

> It actually doesn't look like you use spark context within the predict
> method itself...
>
> Try Removing the spark context ref from the method Params and also the
> (sc) at the end of the predict method.
>
> On Thu, Oct 5, 2017 at 10:13 PM Shane Johnson <
> shanewaldenjohnson@gmail.com> wrote:
>
>> Thanks Daniel. I had the P2Algorithm working before I had to query the
>> EventStore within the predict method. Do you think this is still the issue
>> with the context that I had it working before attempting to add the
>> SparkContext?
>>
>> *Shane Johnson | 801.360.3350 <(801)%20360-3350>*
>> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook
>> <https://www.facebook.com/shane.johnson.71653>
>>
>> On Thu, Oct 5, 2017 at 3:07 PM, Daniel O' Shaughnessy <
>> danieljamesdavid@gmail.com> wrote:
>>
>>> Hi Shane,
>>>
>>> Your RFAlgorithm class needs to use PAlgorithm instead of P2Algorithm.
>>> You then need to write some code to save and load your model and spark
>>> context etc.
>>>
>>> There should be examples of this on the predictionio site somewhere
>>>
>>> On Thu, Oct 5, 2017 at 10:00 PM Shane Johnson <
>>> shanewaldenjohnson@gmail.com> wrote:
>>>
>>>> Hi team,
>>>>
>>>> Can someone guide how I can add SparkContext into the predict method. I
>>>> am using unique ids that I gather from Query and pulling back additional
>>>> attributes from the PEventStore and am getting an error that "sc" cannot
be
>>>> found. When I add SparkContext to the method I get the following error.
>>>>
>>>> Can anyone provide direction here?
>>>>
>>>> Thank you
>>>>
>>>> Adding SparkContext eliminates the first error but produces another.
>>>>
>>>>   def predict(sc: SparkContext, model: RFModel, query: Query):
>>>> PredictedResult = {
>>>>
>>>>     val featureIndex = model.featureIndex
>>>>     val featureCategoricalIntMap = model.featureCategoricalIntMap
>>>>
>>>>
>>>>     val responses: List[PredictionResponses] = query.predictionRequests
>>>>     .map {
>>>>         Predictions =>
>>>>
>>>>             val oppPost = PEventStore.aggregateProperties(
>>>>               appName = sys.env("PIO_EVENTSERVER_APP_NAME"),
>>>>               entityType = "Opportunity"
>>>>             )(sc)
>>>>
>>>>
>>>> First error when sc: SparkContext is not added to the method parameters:
>>>>
>>>> not found: value sc
>>>> [INFO] [Engine$] [error]             )(sc)
>>>> [INFO] [Engine$] [error]               ^
>>>>
>>>>
>>>> Error after adding SparkContext:
>>>>
>>>> class RFAlgorithm needs to be abstract, since method predict in class
>>>> P2LAlgorithm of type (model: org.template.liftscori
>>>> ng.RFModel, query: org.template.liftscoring.Query)org.template
>>>> .liftscoring.PredictedResult is not defined
>>>> [INFO] [Engine$] [error] class RFAlgorithm(val ap: RFAlgorithmParams)
>>>> [INFO] [Engine$] [error]       ^
>>>> [INFO] [Engine$] [error] one error found
>>>> [INFO] [Engine$] [error] (compile:compileIncremental) Compilation
>>>> failed
>>>> [INFO] [Engine$] [error] Total time: 6 s, completed Oct 5, 2017 2:44:51
>>>> PM
>>>>
>>>> *Shane Johnson | 801.360.3350 <(801)%20360-3350>*
>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook
>>>> <https://www.facebook.com/shane.johnson.71653>
>>>>
>>>
>>

Mime
View raw message