incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Sante <tom.sa...@gmail.com>
Subject Re: couchdb for genome data
Date Thu, 04 Mar 2010 09:08:57 GMT
Indeed using "type" could be the alternative to mongoDB collections.  
But my question is if I have billions of documents in the DB would  
this make view generation very slow and take up lot of disk space just  
to be able to search for all probes with a certain experiment_id. Like  
I said the data is structured in experiments so almost all queries and  
changes to the data will be within an experiment with no need to act  
on the huge amount of probes from the other experiments.
Thanks,
Tom

Op 4-mrt-2010 om 08:35 heeft km <srikrishnamohan@gmail.com> het  
volgende geschreven:
> Hi,
>
> You could have an additional key in the document identifying it as  
> probe -
> eg "type" (key) with value  "probe" like this:
>
> {
>       "type":"probe".
>       "probe_id" : 1234567890,
>       "experiment_id" : 1234567890,
>       "raw_value" : 0.43524,
>       "analysis": { "cbs" : 0.436, "CBS+GLAD" : 0.4356 }
> }
>
> so all your probe documents would contain a key called  "type" set to
> "probe". you can identify only these documents with this key.
> Now when u design a view to search probe documents alone, u could  
> use a
> simple filter statement like this:
> if(doc.type=='probe'){ do something ...}
> this will only search/index probe type documents.
>
> NOTE: "type" is not a user defined key just like any other key - u  
> can use
> anyother name for it !
>
> U might have other types of documents for which the type keyword  
> will differ
> accordingly.
> Here there is no need to explicitly define a collection as in Mongodb.
> All JSON documents could be stored in a single database.
>
> HTH,
> Krishna
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> On Thu, Mar 4, 2010 at 7:21 AM, Tom Sante <tom.sante@gmail.com> wrote:
>
>> Hi
>>
>> The data is now stored in a mysql table with about a billion (1000  
>> million)
>> rows.
>> These rows are the data of a genetic test (arrayCGH) and build up  
>> like
>> this:
>>
>> Every experiment (a few thousand of them total) contains  
>> measurements of
>> about 180000 genetic probes. This raw data will be analyzed and the  
>> values
>> run through different algorithms, so every probe needs to store  
>> more than 1
>> value after the analysis is done. The values of different analysis  
>> are now
>> stored in columns in that table making it a pain if we have to add a
>> analysis to the table not yet part of the existing columns. This is  
>> why a
>> schema free document based DB is probably a better fit.
>> The initial idea was to give each probe a separate document, and  
>> when the
>> original value is transform to an other value store this in the same
>> document.
>>
>> {
>>       "probe_id" : 1234567890,
>>       "experiment_id" : 1234567890,
>>       "raw_value" : 0.43524,
>>       "analysis": { "cbs" : 0.436, "CBS+GLAD" : 0.4356 }
>> }
>>
>> Once added to the database almost all changes to the data will be  
>> contained
>> within an experiment.
>>
>> MongoDB has something like collections that would be a appropriate
>> abstraction ~ experiment. But in couchdb I would have to add all  
>> these probe
>> documents in 1 big database without collections. So if I only make  
>> changes
>> to probes within an experiment this would influence the views of  
>> all the
>> other billions document in the db. Because of the large number of  
>> documents
>> it would be good to know beforehand what the implications are of this
>> performance wise?
>>
>> Any suggestions are welcome.
>>
>> Tom
>>

Mime
View raw message