Upgrade Config Server to Replica Set

Share us

Upgrade Config Servers to Replica Set

Only upgrade Mirrored config servers (SCCC) to CSRS if all members of the mongod config instance are at least v3.2.4.

The existing config server must be in sync.

To check config server is in sync or not-

We shouldn’t get mongod config logs like-

ERROR: config servers not in sync! config servers <healthy-server> and <out-of-sync-server> differ

 

Considerations

The upgrade process does not require any downtime. However, while you upgrade the sharded cluster, ensure that clients do not make changes to the collection meta-data. For example, during the upgrade, do not do any of the following:

Take a backup of the config database before upgrading the sharded cluster.

Config server:

server1:27000
server2:27000
server3:27000
Step 1) Disable the Balancer

Login to MongoS instance and disable the Balancer

mongos> sh.getBalancerState()

true

mongos> sh.isBalancerRunning()

false

mongos> sh.stopBalancer()

Waiting for active hosts…

Waiting for the balancer lock…

Waiting again for active hosts after balancer is off…

mongos> sh.getBalancerState()

false

mongos>

 

Step 2) Initiate the Replica Set-

MongoS config file with ConfigDB configuration-

configdb = server1:27000,server2:27000,server3:27000

 

  • The first config server refer to server1:27000
  • The second config server refer to server2:27000
  • The third config server refer to server3:27000

Connect to mongo shell to the first config server listed in the configDB setting of the mongos. In our case it’s server1:27000. And run rs.initiate() to initiate the single member replica set.

rs.initiate( { _id: “csReplSet”, configsvr: true, version: 1, members: [ { _id: 0, host: “server1:27000} ] } )

 

Step 3) Restart this config server as a single member Replica Set with-

  • the –replSet option set to the replica set name specified during the rs.initiate(),
  • the –configsvrMode option set to the legacy config server mode Sync Cluster Connection Config (sccc),
  • the –configsvr option,
  • the –storageEngine option set to the storage engine used by this config server. For this upgrade procedure, the existing config server can be using either MMAPv1 or WiredTiger, and
  • the –port option set to the same port as before restart, and
  • the –dbpath option set to the same path as before restart.

 

Use the below command to add the parameter configsvr, configsvrMode and replSet in the config file of config-db.

sed -i “/configsvr = true\|configsvr=true\|configsvr =true\|configsvr= true/a  configsvrMode = sccc\n\nreplSet = csReplSet\n\n” /opt/mongo/this_host/admin/conf/mongod-configdb.conf

 

configsvr = true

replSet = csReplSet

configsvrMode = sccc

 

Step 3) Start the new mongod instances to add to the replica set.

These instances must use the WiredTigerstorage engine. Starting in 3.2, the default storage engine is WiredTiger for new mongod instances with new data paths.

  • Do not add existing config servers to the replica set.
  • Use new dbpaths for the new instances.

The number of new mongod instances to add depends on the config server currently in the single-member replica set:

  • If the config server is using MMAPv1, start 3 new mongod instances.
  • If the config server is using WiredTiger, start 2 new mongod instances.

In this upgrade procedure, we assumes that the existing config servers use MMAPv1 storageEngine and 3 config servers.

 

Start 3 new MongoD config server instance with different db-path, port no, logpath, configsvr, replSet and unixSocketPrefix. We are adding unixSocketPrefix as we don’t have permission to create new socket file in /tmp. Sample of Config-server db file is given below.

cat /opt/mongo/admin/conf/mongod-configdb-replset.conf

dbpath=/opt/mongo/admin/data/configsvr_replset

pidfilepath = /opt/mongo/admin/data/configsvr_replset/mongod.pid

logpath = /opt/mongo/admin/logs/mongod-config-replset.log

logappend = true

fork = true

#rest = true

port = 27006

configsvr = true

replSet = csReplSet

auth = true

keyFile = /opt/mongo/admin/conf/mongo.key

#mms-token = <token>

#mms-name = <server-name>

#mms-interval = <seconds>

unixSocketPrefix=/opt/mongo

 

And then start the mongod config server.

/opt/mongo/bin/mongod -f /opt/mongo/admin/conf/mongod-configdb-replset.conf

 

Step 4) Using the mongo shell connect to primary replica set ( server1 ) and add the new mongod config instance as non-voting, priority 0 members :

rs.add( { host: “server2:27005”, priority: 0, votes: 0 } )

 

rs.add( { host: “server2:27006”, priority: 0, votes: 0 } )

rs.add( { host: “server3:27007”, priority: 0, votes: 0 } )

 

Step 5) Connect to Mongo shell of primary replica set and make sure these members have completed initial sync and reached the secondary state.

rs.status()

 

Step 6) Shut down one of the other non-replica set config server- i.e. either the second and third config server(server2:27000, server3:27000 ) listed in the configDB setting of the mongos. At this point the config servers will go read-only, meaning certain operations – such as creating and dropping databases and sharded collections – will not be available.

Step 7) Re-configure the replica to allow all members to vote and have default priority of 1. Connect to primary member of CSRS(server1:27000)

var cfg = rs.conf();

cfg.members[0].priority = 1;

cfg.members[1].priority = 1;

cfg.members[2].priority = 1;

cfg.members[3].priority = 1;

cfg.members[0].votes = 1;

cfg.members[1].votes = 1;

cfg.members[2].votes = 1;

cfg.members[3].votes = 1;

rs.reconfig(cfg);

 

Step 7) Step down the first config server of the replica set (server1:27000 ) i.e. the server started with –configsvrMode=sccc.

rs.stepDown(600)

 

Step 7) Shut down the first config server (server1:27000 )

Get the process id and kill it.

$ kill process_id

 

Step 8) Re-start the first config server without sccc.

We can use same db-path, port-no, log-file in config file and remove configsvrMode = sccc or if we want to use different config file then we can do that as well.

rs.stepDown(600)

sed -i “/configsvrMode = sccc/d” /opt/mongo/this_host/admin/conf/mongod-configdb.conf

/opt/mongo/bin/mongod -f /opt/mongo/admin/conf/mongod-configdb.conf

 

 

 

If the first config server uses the MMAPv1 storage engine, the member will transition to “REMOVED“state.

At this point the config server data will return to being writeable and all catalog operations – including creating and dropping databases and sharded collections – will once again be possible.

 

Step 9) Re-start the MongoS Instances with updated configDB option in MongoS config file /opt/mongo/upgradeRep1A/rs1/admin/conf/mongos.conf.

configdb = csReplSet/server1:27005,server2:27006,server3:27007

 

kill the MongoS instance and restart with config file.

$ kill process_id

/opt/mongo/bin/mongos -f /opt/mongo/upgradeRep1A/rs1/admin/conf/mongos.conf

 

Step 10) Verify that the restarted mongos instances are aware of the protocol change. Connect a mongo shell to a mongos instance and check the mongos collection in the config database:

The ping value for the mongos instances should indicate some time after the restart.

use config

db.mongos.find()

{ “_id” : “server2:20000″, “ping” : ISODate(“2018-04-23T09:58:51.160Z”), “up” : NumberLong(1118358), “waiting” : true, “mongoVersion” : “3.2.8” }

{ “_id” : “server3:20000″, “ping” : ISODate(“2018-04-23T10:02:11.180Z”), “up” : NumberLong(1120091), “waiting” : true, “mongoVersion” : “3.2.8” }

{ “_id” : “server1:20000″, “ping” : ISODate(“2018-04-23T10:03:23.098Z”), “up” : NumberLong(1196788), “waiting” : true, “mongoVersion” : “3.2.8” }

 

Step 11) If the first config server uses the MMAPv1 storage engine, remove the member from the replica set. Connect a mongo shell to the current primary and use rs.remove()

rs.remove(”server1:27000″)

 

Step 11) shut down the remaining non-replica set config server.

Step 12) Re-enable the balancer.

Re-enable the balancer from MongoS shell against admin database.

mongos> sh.getBalancerState()

false

mongos> sh.startBalancer()

mongos> sh.getBalancerState()

true

mongos>

 

Hope, it will help.  Please Share your inputs and feedback.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.