Procedure - (Node + MongoDB) Application Performance BenchMarking with Jmeter + Grafana + collectd

Well, the post subject is very illustrative ...

How do you benchmark your app/database tiers?, there is a lot of tools to do that, you just have to search for those procedures that fits your needs.

Our typicall test environment is like this:

In Sepalo Software we are using Jmeter as our reference benchmarking tool, most of the apps can be stressed by API calls that we can easy generate with Jmeter. Apps are typically Node apps (Cloud based) with MongoDB (Cloud based) as backend database.

Our first goal was to parametrize all the api calls scenarios by grouping them and generating a lot of parameters that can be passed to the jmeter executable to modify the benchmark scenario as we need (more update calls vs insert calls, ramp users, test interval, burst intervals, and so on) and even the type of scenario itself (Stability, Stress, Burst).

We met this goal by using jmeter parameters (e.g." ${__P(users,1)}" ) that we can specify in command line jmeter executable call.

With these parameters we can execute multiple test scenarios:

• Stability Test scenario that is mean to verify if application can continuously perform well within or just above the acceptable period. It is often referred as load or Endurance testing.
• Burst Test scenario is a type of performance testing focused on determining an application’s robustness, availability, and reliability under peak load conditions. The goal of burst testing is to check if application performance recovers after the peak conditions.
• Stress Test scenario is a type of performance testing focused on determining an application’s robustness, availability, and reliability under extreme conditions. The goal of stress testing is to identify application issues that arise or become apparent only under extreme conditions.

An also, we can have multiple different configurations within a single scenario (distinct percentage api calls of behaviors of each request).

After these automated and configurable jmeter benchmarks test, we realized that most of our test time was focused in the test data results preparation, where we spend time taking csv outputted files and making spreadsheets dynamic tables and graphics.

So we just configured Grafana as our time dasboard based solution to show the tests results, which are saved automatically by jmeter in a influxDB database. For OS statistics and metrics, we use collectd that automatically save all servers statistics in the influxDB. All these tools are open source (which does not mean free).

Finally, we have customizable web based results like these

With this test environment configuration (Cloud based of course ...) we are spending our time in the important phase of the whole process, the initial API calls definitions to stress the apps.

Please, feel free to contact with Sepalo Software if you need more information.

Procedure - MongoDB Rename with minimal interruption

Rename a mongoDB database can be a difficult process because this operation will requiere a database outage, you have to make a backup of whole database, create the new database and restore into it. (mongodump, mongoimport) or by db.copyDatabase() command.

Those commands don´t take care of online operations, so, you need a database blackout to perform a database rename.

In Sepalo Software we have designed a procedure to overcome this issue, by using a mix of incremental database recovery and database rename, with only some seconds of outage.

The procedure is based in incremental backups (oplogs) and incremental restores into a new database (the renamed one) doing a previous oplogs entries substitution to rename namespaces.

Please, feel free to contact with Sepalo Software if you need more information.

Script - MongoDB stats, execution plans and clearing

Some usefull methods to review and clear execution plans in MongoDb

MongoDB stats, collections stats

db.stats()
db.<collection>.stats()

Execution plans clearing, needed if an incorrect execution plan appears and you need to refresh them.

db.<collection>.getPlanCache().clear()

Show execution plans by collection or by query

db.<collection>.getPlanCache().listQueryShapes()
db.<collection>.getPlanCache().getPlansByQuery({
        "query" : {
            "$or" : [
                {
                    "key1" : "value1",
                    "key2" : "value2"
                }
            ],
            "key3" : "value3"
        },
        "sort" : {},
        "projection" : {}
    })

Script - MongoDB online data refresh method

Here we have an usefull command to refresh data from a mongoDB database to another.
No extra space is required (no backup space required, it will be done on the fly by a pipe command).

mongodump --host <host_orig> --username=<user_orig> --password=<password_orig> --authenticationDatabase=admin -d <db_orig> --archive --numParallelCollections=2| mongorestore --host=<host_dest> --username=<user_dest> --password=<password_dest> --authenticationDatabase=admin --nsFrom=<db_orig>.* --nsTo=<db_dest>.* --archive --numParallelCollections 2 --numInsertionWorkersPerCollection 2 2>&1

Where

<host_orig> = Original server ip address
<user_orig> = Original username with backup privileges
<password_orig> = Original username password
<db_orig> = Original database name


<host_dest> = Destination server ip address
<user_dest> = Destination username with restore privileges
<password_dest> = Destination username password
<db_dest> = Destination database name (can be different from original database name)

We can use this data refresh method in multiple situations, by example, if we want to rename a database, or create a dev environment from production, ...

Páginas a la Carta - Lectópolis

El 8 de sept a partir de las 14:00 se abren las votaciones de la Plataforma Talento de ElPais.com, si te gusta la idea, no dudes en votarla en el siguiente enlace: https://t.co/1WTvLi7Kh6 (debes registrarte previamente en elpais.com).

gracias!!!

Script - PostgreSQL multiple async parallel execution in PL/pgSQL

If you need to split and parallelize a pl/pgsql procedure execution to get better performance (due to the fact that a single pl/pgsql executes only in 1 thread cpu, it´s constraint to 1 single thread cpu), you can try the use of dblink async calls, which will save you a lot of time, without need to do more complicated solutions.

By example, we need to do a  massive insert into a table with a insert as select query, the server has a lot of IO capacity that isn´t used by a single pl/pgsql execution. Having a method to do a partition (by index date ranges or similar), you could do this:

-- Create the dblink extension
CREATE EXTENSION dblink;
GRANT EXECUTE ON FUNCTION dblink_connect_u(text) TO <user>;
GRANT EXECUTE ON FUNCTION dblink_connect_u(text, text) TO <user>;

select dblink_connect_u('dblink1','dbname=<database> user=<user>');
select dblink_connect_u('dblink2','dbname=<database> user=<user>');
select dblink_connect_u('dblink3','dbname=<database> user=<user>');



-- These are the aync calls, they will return immediatly and will begin to execute in backend
select * from dblink_send_query('dblink1','INSERT INTO <dest_table> SELECT * FROM <orig_table> WHERE AUD_FEC BETWEEN <FEC_INI1> AND <FEC_FIN1>');
select * from dblink_send_query('dblink2','INSERT INTO <dest_table> SELECT * FROM <orig_table> WHERE AUD_FEC BETWEEN <FEC_INI2> AND <FEC_FIN2>');
select * from dblink_send_query('dblink3','INSERT INTO <dest_table> SELECT * FROM <orig_table> WHERE AUD_FEC BETWEEN <FEC_INI3> AND <FEC_FIN3>');

-- Here we wait for each finished results
select * from dblink_get_result('dblink1') as t1(a text);
select * from dblink_get_result('dblink2') as t1(a text);
select * from dblink_get_result('dblink3') as t1(a text);

SELECT dblink_disconnect('dblink1');
SELECT dblink_disconnect('dblink2');
SELECT dblink_disconnect('dblink3');


If we have a date interval (fec_ini, fec_fin) and orig_table is indexed by aud_fec, we can use this to do an automatic date partition to parallelize up to maxParall variable inside a PL/pgSQL procedure:

SELECT cast((fec_fin-fec_ini)/maxParall as text) INTO vInterval;

FOR numParall IN 1..maxParall LOOP
    select dblink_connect_u('dblink'||numParall,'dbname=database user=user') into vText;
    select * from dblink_send_query('dblink'||numParall,'INSERT INTO dest_table SELECT * FROM orig_table WHERE AUD_FEC >= '||fec_ini||'+'||numParall-1||'*CAST('||chr(39)||vInterval||chr(39)||' AS interval) AND AUD_FEC < '||fec_ini||'+'||numParall||'*CAST('||chr(39)||vInterval||chr(39)||' AS interval)') into vInt;
END LOOP;       
FOR numParall IN 1..maxParall LOOP
    select * from dblink_get_result('dblink'||numParall) as t1(a text) into vText;
    SELECT dblink_disconnect('dblink'||numParall) into vText;
END LOOP;       

Hope this can help you.

Pages on demand

Press release about lectópolis

Related video (spanish)