Indexing the whole db

[slaniado]

i want to index all the db, is there a way to do that or i need to create a river for each collection

[richardwilly98]

You need to create a river for each collection.

On Thursday, November 21, 2013, Saul Laniado wrote:

i want to index all the db, is there a way to do that or i need to create
a river for each collection

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/177
.

[slaniado]

thank's

[richardwilly98]

Let me know if you think your use case can justify to implement a new feature.
I am not saying it will be implemented :-)

[slaniado]

in my case i have a large number of collections, every time i add new type of document i create a new collection. To resolve this i will need to have a list of the rivers that i already have in ES and every time that i make a new insert in mongo i will need to check if there is the need to create a new river for this collection.
I think that a option to index all the DB in ES is a great idea.
Thank you

[richardwilly98]

I have just release 1.7.3.
you can install using the following command: plugin -i com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/1.7.3
You should use the parameters import_all_collections and maybe also drop_collection see this file [1] from more details.

I am planning to update the wiki by end of this week.

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/blob/530ce26400f802936cc2a5041865757ca32bf90c/manual-testing/issues/177/simple-mongodb-river-import-all-collections.json

[slaniado]

Great, thank’s a lot!

Saul

El 05/12/2013, a las 08:56, Richard Louapre notifications@github.com escribió:

I have just release 1.7.3.
you can install using the following command: plugin -i com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/1.7.3
You should use the parameters import_all_collections and maybe also drop_collection see this file [1] from more details.

I am planning to update the wiki by end of this week.

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/blob/530ce26400f802936cc2a5041865757ca32bf90c/manual-testing/issues/177/simple-mongodb-river-import-all-collections.json

—
Reply to this email directly or view it on GitHub.

[hepr]

Hi,

I would like to use include_all_collection but I've got this log :

[2013-12-17 18:33:21,880][INFO ][org.elasticsearch.river.mongodb.Slurper] Skip initial import from collection mongodb177
[2013-12-17 18:33:21,881][INFO ][org.elasticsearch.river.mongodb.Slurper] No known previous slurping time for this collection

My config is :
{
"type": "mongodb",
"mongodb": {
"db": "mean-dev",
"options": {
"drop_collection": true,
"import_all_collections": true
}
},
"index": {
"name": "idx"
}
}

curl -XPUT "http://localhost:9200/_river/mongodb177/_meta" -d @all_collection.json

MongoDB : 2.4.8
Mongo-River : 1.7.3
Elasticseach : 0.90.7

curl -XGET localhost:9200/idx/articles/_count
{"count":0,"_shards":{"total":5,"successful":5,"failed":0}}

Or there are 3 documents in the database.

Can you help me ?

Thanks

Herve

[richardwilly98]

@hepr
Can you please do the following steps?
Delete the river.
Enable logging [1].
Restart ES
Register the river
Provide full ES log file.

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki#troubleshooting

[hepr]

Hi,

I've done all.
I joined my log file.

[2013-12-17 19:30:45,673][INFO ][node ] [Richards, Susan] version[0.90.7], pid[12378], build[36897d0/2013-11-13T12:06:54Z]
[2013-12-17 19:30:45,673][INFO ][node ] [Richards, Susan] initializing ...
[2013-12-17 19:30:45,681][INFO ][plugins ] [Richards, Susan] loaded [], sites []
[2013-12-17 19:30:47,776][INFO ][node ] [Richards, Susan] initialized
[2013-12-17 19:30:47,777][INFO ][node ] [Richards, Susan] starting ...
[2013-12-17 19:30:47,864][INFO ][transport ] [Richards, Susan] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.11:9300]}
[2013-12-17 19:30:50,958][INFO ][cluster.service ] [Richards, Susan] new_master [Richards, Susan][edpSSFUXTmGQrfXrSebyew][inet[/192.168.0.11:9300]], reason: zen-disco-join (elected_as_master)
[2013-12-17 19:30:50,989][INFO ][discovery ] [Richards, Susan] elasticsearch/edpSSFUXTmGQrfXrSebyew
[2013-12-17 19:30:51,006][INFO ][http ] [Richards, Susan] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.11:9200]}
[2013-12-17 19:30:51,007][INFO ][node ] [Richards, Susan] started
[2013-12-17 19:30:51,034][INFO ][gateway ] [Richards, Susan] recovered [0] indices into cluster_state
[2013-12-17 19:30:55,952][INFO ][node ] [Richards, Susan] stopping ...
[2013-12-17 19:30:56,004][INFO ][node ] [Richards, Susan] stopped
[2013-12-17 19:30:56,005][INFO ][node ] [Richards, Susan] closing ...
[2013-12-17 19:30:56,018][INFO ][node ] [Richards, Susan] closed
[2013-12-17 19:31:21,313][INFO ][node ] [Vindicator] version[0.90.7], pid[12507], build[36897d0/2013-11-13T12:06:54Z]
[2013-12-17 19:31:21,314][INFO ][node ] [Vindicator] initializing ...
[2013-12-17 19:31:21,325][INFO ][plugins ] [Vindicator] loaded [mongodb-river], sites [river-mongodb]
[2013-12-17 19:31:23,302][TRACE][rest.action.mongodb ] [Vindicator] RestMongoDBRiverAction - baseUrl: /_river/mongodb
[2013-12-17 19:31:23,376][INFO ][node ] [Vindicator] initialized
[2013-12-17 19:31:23,377][INFO ][node ] [Vindicator] starting ...
[2013-12-17 19:31:23,472][INFO ][transport ] [Vindicator] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.11:9300]}
[2013-12-17 19:31:26,548][INFO ][cluster.service ] [Vindicator] new_master [Vindicator][bbIdlxNsRIGbINdvgFQxwg][inet[/192.168.0.11:9300]], reason: zen-disco-join (elected_as_master)
[2013-12-17 19:31:26,583][INFO ][discovery ] [Vindicator] elasticsearch/bbIdlxNsRIGbINdvgFQxwg
[2013-12-17 19:31:26,603][INFO ][http ] [Vindicator] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.11:9200]}
[2013-12-17 19:31:26,603][INFO ][node ] [Vindicator] started
[2013-12-17 19:31:26,626][INFO ][gateway ] [Vindicator] recovered [0] indices into cluster_state
[2013-12-17 19:31:56,442][INFO ][cluster.metadata ] [Vindicator] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings []
[2013-12-17 19:31:56,765][INFO ][cluster.metadata ] [Vindicator] [_river] update_mapping mongodb177
[2013-12-17 19:31:56,772][INFO ][river.routing ] [Vindicator] no river _meta document found, retrying in 1000 ms
[2013-12-17 19:31:57,799][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Initializing river : [mongodb177]
[2013-12-17 19:31:57,800][INFO ][river.mongodb ] Parse river settings for mongodb177
[2013-12-17 19:31:57,822][TRACE][river.mongodb ] mongoOptionsSettings: {drop_collection=true, import_all_collections=true}
[2013-12-17 19:31:57,824][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb177
[2013-12-17 19:31:57,825][DEBUG][river.mongodb.util ] setRiverStatus called with mongodb177 - RUNNING
[2013-12-17 19:31:57,829][DEBUG][org.elasticsearch.river.mongodb.MongoDBRiver] Using mongodb server(s): host [localhost], port [27017]
[2013-12-17 19:31:57,831][INFO ][cluster.metadata ] [Vindicator] [_river] update_mapping mongodb177
[2013-12-17 19:31:57,832][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[1.7.3] - hash[4ca377f] - time[2013-12-05T14:34:13Z]
[2013-12-17 19:31:57,832][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [true], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [mean-dev], collection [mongodb177], script [null], indexing to [idx]/[mean-dev]
[2013-12-17 19:31:57,906][INFO ][cluster.metadata ] [Vindicator] [idx] creating index, cause [api], shards [5]/[1], mappings []
[2013-12-17 19:31:58,120][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoAdminUser: - isAuthenticated: false
[2013-12-17 19:31:58,240][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.4.8
[2013-12-17 19:31:58,243][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] serverStatus: { "serverUsed" : "localusr-LIFEBOOK-U772/127.0.1.1:27017" , "host" : "localusr-LIFEBOOK-U772" , "version" : "2.4.8" , "process" : "mongod" , "pid" : 6155 , "uptime" : 11065.0 , "uptimeMillis" : 11064694 , "uptimeEstimate" : 10934.0 , "localTime" : { "$date" : "2013-12-17T18:31:58.230Z"} , "asserts" : { "regular" : 0 , "warning" : 0 , "msg" : 0 , "user" : 28 , "rollovers" : 0} , "backgroundFlushing" : { "flushes" : 184 , "total_ms" : 2397 , "average_ms" : 13.027173913043478 , "last_ms" : 13 , "last_finished" : { "$date" : "2013-12-17T18:31:33.577Z"}} , "connections" : { "current" : 24 , "available" : 15976 , "totalCreated" : 417} , "cursors" : { "totalOpen" : 22 , "clientCursors_size" : 22 , "timedOut" : 1 , "totalNoTimeout" : 22} , "dur" : { "commits" : 30 , "journaledMB" : 0.0 , "writeToDataFilesMB" : 0.0 , "compression" : 0.0 , "commitsInWriteLock" : 0 , "earlyCommits" : 0 , "timeMs" : { "dt" : 3071 , "prepLogBuffer" : 0 , "writeToJournal" : 0 , "writeToDataFiles" : 0 , "remapPrivateView" : 0}} , "extra_info" : { "note" : "fields vary by platform" , "heap_usage_bytes" : 69604368 , "page_faults" : 49} , "globalLock" : { "totalTime" : 11064694000 , "lockTime" : 505464 , "currentQueue" : { "total" : 0 , "readers" : 0 , "writers" : 0} , "activeClients" : { "total" : 0 , "readers" : 0 , "writers" : 0}} , "indexCounters" : { "accesses" : 3105 , "hits" : 3105 , "misses" : 0 , "resets" : 0 , "missRatio" : 0.0} , "locks" : { "." : { "timeLockedMicros" : { "R" : 524473 , "W" : 505464} , "timeAcquiringMicros" : { "R" : 407892 , "W" : 56600}} , "admin" : { "timeLockedMicros" : { "r" : 24189 , "w" : 0} , "timeAcquiringMicros" : { "r" : 586 , "w" : 0}} , "local" : { "timeLockedMicros" : { "r" : 632052 , "w" : 7992} , "timeAcquiringMicros" : { "r" : 119523 , "w" : 5314}} , "ffxi" : { "timeLockedMicros" : { "r" : 4583 , "w" : 0} , "timeAcquiringMicros" : { "r" : 461 , "w" : 0}} , "chaumeil-new" : { "timeLockedMicros" : { "r" : 22663 , "w" : 20505} , "timeAcquiringMicros" : { "r" : 1089 , "w" : 970}} , "europexpress" : { "timeLockedMicros" : { "r" : 19677 , "w" : 0} , "timeAcquiringMicros" : { "r" : 895 , "w" : 0}} , "test" : { "timeLockedMicros" : { "r" : 5671 , "w" : 0} , "timeAcquiringMicros" : { "r" : 451 , "w" : 0}} , "user-auth" : { "timeLockedMicros" : { "r" : 12007 , "w" : 0} , "timeAcquiringMicros" : { "r" : 702 , "w" : 0}} , "mean-dev" : { "timeLockedMicros" : { "r" : 16272 , "w" : 17218} , "timeAcquiringMicros" : { "r" : 858 , "w" : 879}} , "flydb" : { "timeLockedMicros" : { "r" : 12761 , "w" : 0} , "timeAcquiringMicros" : { "r" : 866 , "w" : 0}} , "files" : { "timeLockedMicros" : { "r" : 14722 , "w" : 198} , "timeAcquiringMicros" : { "r" : 852 , "w" : 10}} , "chaumeil" : { "timeLockedMicros" : { "r" : 264994 , "w" : 146859} , "timeAcquiringMicros" : { "r" : 124511 , "w" : 100522}}} , "network" : { "bytesIn" : 2669693 , "bytesOut" : 36197057 , "numRequests" : 41769} , "opcounters" : { "insert" : 23 , "query" : 3987 , "update" : 88 , "delete" : 4 , "getmore" : 2121 , "command" : 37513} , "opcountersRepl" : { "insert" : 0 , "query" : 0 , "update" : 0 , "delete" : 0 , "getmore" : 0 , "command" : 0} , "recordStats" : { "accessesNotInMemory" : 15 , "pageFaultExceptionsThrown" : 13 , "admin" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "chaumeil" : { "accessesNotInMemory" : 13 , "pageFaultExceptionsThrown" : 13} , "chaumeil-new" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 0} , "europexpress" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "files" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "flydb" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "local" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "mean-dev" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 0} , "user-auth" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0}} , "repl" : { "setName" : "symeos" , "ismaster" : true , "secondary" : false , "hosts" : [ "localusr-LIFEBOOK-U772:27017"] , "primary" : "localusr-LIFEBOOK-U772:27017" , "me" : "localusr-LIFEBOOK-U772:27017"} , "writeBacksQueued" : false , "mem" : { "bits" : 64 , "resident" : 65 , "virtual" : 11653 , "supported" : true , "mapped" : 4734 , "mappedWithJournal" : 9468} , "metrics" : { "document" : { "deleted" : 1 , "inserted" : 23 , "returned" : 22221 , "updated" : 88} , "getLastError" : { "wtime" : { "num" : 0 , "totalMillis" : 0} , "wtimeouts" : 0} , "operation" : { "fastmod" : 0 , "idhack" : 1898 , "scanAndOrder" : 31} , "queryExecutor" : { "scanned" : 35217} , "record" : { "moves" : 4} , "repl" : { "apply" : { "batches" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0} , "buffer" : { "count" : 0 , "maxSizeBytes" : 268435456 , "sizeBytes" : 0} , "network" : { "bytes" : 0 , "getmores" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0 , "readersCreated" : 3} , "oplog" : { "insert" : { "num" : 688 , "totalMillis" : 0} , "insertBytes" : 403927} , "preload" : { "docs" : { "num" : 0 , "totalMillis" : 0} , "indexes" : { "num" : 0 , "totalMillis" : 0}}} , "ttl" : { "deletedDocuments" : 0 , "passes" : 184}} , "ok" : 1.0}
[2013-12-17 19:31:58,243][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] process: mongod
[2013-12-17 19:31:58,254][INFO ][org.elasticsearch.river.mongodb.Slurper] Skip initial import from collection mongodb177
[2013-12-17 19:31:58,255][INFO ][org.elasticsearch.river.mongodb.Slurper] No known previous slurping time for this collection
[2013-12-17 19:31:58,265][INFO ][cluster.metadata ] [Vindicator] [_river] update_mapping mongodb177

[richardwilly98]

@hepr
I suspect that isValidOplogEntry return false.
Can you please confirm oplog.rs collection contains entries with the correct namespace?

If oplog.rs is not too big can you please dump it here?
Or you could also filter it by ns starts with mean-dev

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/blob/elasticsearch-river-mongodb-1.7.3/src/main/java/org/elasticsearch/river/mongodb/Slurper.java#L360

[hepr]

My oplog.rs is too big.
See below a part of it.
I've done a flter on ns with mean-dev but it found no record.

/* 0 */
{
"ts" : Timestamp(1387294357, 1),
"h" : NumberLong(0),
"v" : 2,
"op" : "n",
"ns" : "",
"o" : {
"msg" : "initiating set"
}
}

/* 1 */
{
"ts" : Timestamp(1387294391, 1),
"h" : NumberLong(9058808299029266449),
"v" : 2,
"op" : "i",
"ns" : "chaumeil.system.indexes",
"o" : {
"ns" : "chaumeil.tmp.mr.DolibarrModules_0_inc",
"key" : {
"0" : 1
},
"name" : "0_1"
}
}

[richardwilly98]

@hepr
Can you try to add one more document in mean-dev database while the river is running?
That should generate a new entry in oplog.rs collection and the document should be available in ES.
Please provide ES log file.