hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Help designing application architecture
Date Thu, 07 Jul 2016 23:54:55 GMT
For 1) you don't have to introduce external storage.

You can define case classes for the known formats.

FYI

On Thu, Jul 7, 2016 at 4:40 PM, venito camelas <robotirlandes@gmail.com>
wrote:

> I'm pretty new to this and I have a use case I'm not sure how to
> implement, I'll try to explain it and I'd appreciate if anyone could point
> me in the right direction.
>
> The case has these requirements:
>  1 - Any user shoud be able to define the format of the information they
> want to store (channel). For example, user X defines a channel named
> "coordinate":
> coordinate = {
> "X" : "Float",
> "Y" : "Float",
> "instant" : "Timestamp"
> }
>   Every channel has some time value, it can be an instant (like above) or
> a period of time ("start" : "Timestamp", "end" : "Timestamp")
>
>  2 - Given the previous example, the user should be able to ask the
> following questions:
> 2.1 When was the last time I went near {X : x, Y : y}?  --> Process the
> information in order to get the "near" places and return the newest one.
> 2.2 Where was I on march 6th between 1pm and 2pm?       --> Query by time
>
>
>
> For 1) I was thinking of using some Document oriented storage because of
> the channels lack of structure, not sure that's the only thing to consider
> though.
>
> For 2.1) I'd use some MR job
>
> For 2.2) I think it would be better to have the information in the
> document storage and make the queries there.
>
> Is it a good approach to have the information stored both in the hdfs and
> the document oriented storage (for processing and querying respectively)?
>
> As I mentioned in the beginning, I'm really new to this and I'm just
> trying to learn..so sorry if my doubts are silly.
>
> Any suggestion or any good reference related to this will be much
> appreciated.
>

Mime
View raw message