Mongo Replicas Things

Serveurs configurations

1 server for all replicas (data, avatar, upload)

• Name : mongo-replica-real-time
• Zone : europe-west1-d
• Type : n1-standard-1 (1 vCPU, 3.75 GB memory)
• Disks (standard persisten disks):
• realtime-upload-replica 1000Go -> /data-realtime-upload-replica
• realtime-data-replica 300Go -> /data-realtime-data-replica
• realtime-upload-avatar 200Go -> /data-realtime-avatar-replica
• replica-swap 40Go -> /swap (6 mongos with indexes need a lot of ram)

Process

Service Supervision

Manage with runit.

Examples :

• Check arbiter :

  sudo sv status mongo_data_arbiter

• Start arbiter :

  sudo sv start mongo_data_arbiter

• Stop arbiter :

  sudo sv stop mongo_data_arbiter

• services names : "mongo_avatar_replica" "mongo_data_arbiter" "mongo_data_replica" "mongo_upload_replica" etc...

Files

Mongo binaries	Mongo binaries (current version)
/mongo	/mongo/current

Mongo conf files	Mongo conf (current version)
/mongo/conf	/mongo/current_conf

We need 8 mongo instances :

• secondaryIndexPrefetch is set to _id_only (do not load index to memory except _id)
• the data replica isn't stored with engine: wiredTiger like in production, because "secondaryIndexPrefetch" doesn't works with wiredTiger.

#arbiter_example.conf
net:
  port: 3000
replication:
  replSetName: az_res
storage:
  dbPath: /data-arbiter
  journal:
    enabled: false
  mmapv1:
    smallFiles: true

#avatar_replica.conf
net:
  port: 27117
replication:
  replSetName: avatar_replica
  secondaryIndexPrefetch: _id_only
storage:
  dbPath: /data-realtime-avatar-replica

#data_replica.conf
net:
  port: 27017
replication:
  replSetName: az_rep
  secondaryIndexPrefetch: _id_only
storage:
  dbPath: /data-realtime-data-replica
  engine: wiredTiger

#upload_replica.conf
net:
  port: 27217
replication:
  replSetName: upload_replica
  secondaryIndexPrefetch: _id_only
storage:
  dbPath: /data-realtime-upload-replica

Replica configurations

For each mongo replica we start 2 instances :

• the first is the replica
• the second is an arbiter

Mongo production instance:

• Restart with --replSet az_rep
• Then initialise the replica

rs.initiate()
rs.conf() // to check conf
rs.status() // replica status

• Start the replica sudo sv start mongo_data_replica and the arbiter...
• Add the new instance in the production mongo

rs.add("mongo-replica-real-time:27017")
// cfg = rs.conf()
// rs.isMaster({ isMaster: 1 })
// rs.reconfig({}, {force: true})
rs.addArb("mongo-replica-real-time:3000")
cfg = rs.conf()
cfg.members[2].hidden = true
cfg.members[2].priority = 0
rs.reconfig(cfg)

• Hidden : "A hidden member maintains a copy of the primary’s data set but is invisible to client applications." More informations here.
• Priority 0 : A priority 0 member is a secondary that cannot become primary. More informations here
• Check data replication on the replica :
• connect to local instance : ./mongo
• You should see az_rep:SECONDARY> in the console
• Enter db.getMongo().setSlaveOk() to force replica to accept read operation in the console
• show dbs, use az_app_2_production, db.tasks.count() etc...

How to restore a primary

• Stop all az_app_2 process servers, workers...
• Set the maintenance page on nginx
• Stop primary and secondary instances
• Snapshot both disks (primary and secondary)
• On the primary delete all mongo data files
• Start primary and secondary instances without --replSet option
• On the primary :

//copy db
db.copyDatabase('az_app_avatar_production', 'az_app_avatar_production', 'pmongodb2:27117')

• When the copy is ok restart every thing
• Check if everything work well
• Remove nginx maintenance page
• After the crisis don't forget to reset the replica.

Other Things

Reconfigure a Replica (Replace it, hard way)

• Stop the mongo
• Remove --replSet option
• Restart
• use local
• var doc = db.system.replset.findOne()
• modify doc
• db.system.replset.save(doc)
• db.system.replset.remove({_id:'OldReplicaSetName'})
• Restart with replSet...

Force a SECONDARY to become PRIMARY

• open a mongo shell on the secondary
• db.getMongo().setSlaveOk()
• cfg = rs.conf()
• cfg.members = [cfg.members[<>]]
• rs.reconfig(cfg, {force: true})