Mostrando entradas con la etiqueta MONGODB. Mostrar todas las entradas
Mostrando entradas con la etiqueta MONGODB. Mostrar todas las entradas

martes, 6 de febrero de 2018

Script - MongoDB incremental update preparation

Quick test, please answer:

How do you get prepared for a MongoDB incremental update?

    a) No special preparation, because I trust ... (developer´s team, QA´s team, cloud backups/snapshots, myself if I prepared the update, ...)
    b) With previous backup of the whole database
    c) With previous backup of items to be changed by the update

Well, if your answer is "a", good luck, you will need it, although you have an extremely powerful hardware provider or solution (Cloud, on-premise) with multiple recovery options (snapshots, replication, ...) or even good working database auditing options (with mongo those options comes with the paid version of the engine), it could be difficult to rollback a specific incremental update, moreover if new "good" transactions come after the point of the incremental update. Most of those options will require additional work and resources to be practical for our rollback required scenario.

Everyone makes mistakes, (even the good DBAs ...), so, you should be prepared for these situations.
   
The best approach should be the "b" and "c" options, if you can afford to make a full backup previously to any incremental update, please, do it, but this isn´t always possible because the time and resources consuming of a whole database backup.

Another option, could be to have incremental backups, so you could recover an initial photo of your database and go ahead until the time of the incremental update.

Apart from those options, you could have a simple script to make a pre-backup of the docs to be changed by the update.

This will allow you another way to recover (plus the backups one as mentioned early) in case of an update rollback required.

In Sepalo Software, additionally to periodic backups, incremental backup, cloud snapshot and backup options, we use an automated version of the following script, which is integrated in our devops cycle (automatic incremental update executions, database automatic versioning, ...):
   
-- Pre-backup

limit=0;
skip=0;
sort={};
filt={};
coll="test";
tag="TICKET-123"
db.getCollection(coll).find(filt).sort(sort).skip(skip).limit(limit).forEach(function(x){
    db.updateBackups.insert({"id":tag, "collection": coll, "date":  new Date(), "data":x});
});   
print('Prepared backups of collection:'+coll+',with filter:'+filt+',with identification:'+tag);


-- Rollback

tag="TICKET-123"
db.updateBackups.find({"id":tag}).forEach(function(x){
    db.getCollection(x.collection).remove({"_id":x.data._id},{"multi":false});
    db.getCollection(x.collection).insert(x.data);
});   
print('Rollbacked partial backup with identification:'+tag);


So, for an incremental update like this:

db.users.update({type:"preAuthorized"},{$set:{type:"authorized"}},{multi:true});

Developers should generate a RFC script (id ticket: PRJ-1234) like this:

//Pre-backup
limit=0;
skip=0;
sort={};
filt={type:"preAuthorized"};
coll="users";
tag="PRJ-1234"
db.getCollection(coll).find(filt).sort(sort).skip(skip).limit(limit).forEach(function(x){
    db.updateBackups.insert({"id":tag, "collection": coll, "date":  new Date(), "data":x});
});   
print('Prepared backups of collection:'+coll+',with filter:'+filt+',with identification:'+tag);
//update
db.users.update({type:"preAuthorized"},{$set:{type:"authorized"}},{multi:true});


In case the incremental update needs to be rolled back, we´ll just need to execute:

tag="PRJ-1234"
db.updateBackups.find({"id":tag}).forEach(function(x){
    db.getCollection(x.collection).remove({"_id":x.data._id},{"multi":false});
    db.getCollection(x.collection).insert(x.data);
});   
print('Rollbacked partial backup with identification:'+tag);

That will recover old docs affected by the incremental update scripts.

With this type of scripts, we have an automated and quick way to provide "rollback" script to our incremental updates.

Please feel free to contact Sepalo Software if you want more information about devops automation, increase performance of these scripts (bulk options, parallelizing) or database automatic versioning.


lunes, 29 de enero de 2018

Procedure - MongoDB Rename with minimal interruption

Rename a mongoDB database can be a difficult process because this operation will requiere a database outage, you have to make a backup of whole database, create the new database and restore into it. (mongodump, mongoimport) or by db.copyDatabase() command.

Those commands don´t take care of online operations, so, you need a database blackout to perform a database rename.

In Sepalo Software we have designed a procedure to overcome this issue, by using a mix of incremental database recovery and database rename, with only some seconds of outage.

The procedure is based in incremental backups (oplogs) and incremental restores into a new database (the renamed one) doing a previous oplogs entries substitution to rename namespaces.

Please, feel free to contact with Sepalo Software if you need more information.

jueves, 9 de noviembre de 2017

Script - MongoDB online data refresh method

Here we have an usefull command to refresh data from a mongoDB database to another.
No extra space is required (no backup space required, it will be done on the fly by a pipe command).

mongodump --host <host_orig> --username=<user_orig> --password=<password_orig> --authenticationDatabase=admin -d <db_orig> --archive --numParallelCollections=2| mongorestore --host=<host_dest> --username=<user_dest> --password=<password_dest> --authenticationDatabase=admin --nsFrom=<db_orig>.* --nsTo=<db_dest>.* --archive --numParallelCollections 2 --numInsertionWorkersPerCollection 2 2>&1

Where

<host_orig> = Original server ip address
<user_orig> = Original username with backup privileges
<password_orig> = Original username password
<db_orig> = Original database name


<host_dest> = Destination server ip address
<user_dest> = Destination username with restore privileges
<password_dest> = Destination username password
<db_dest> = Destination database name (can be different from original database name)

We can use this data refresh method in multiple situations, by example, if we want to rename a database, or create a dev environment from production, ...