[MongoDB] 샤딩+레플리카셋 구성 (Sharding with ReplicaSet)

MongoDB 2.6 버전 기준.

서버 5대를 이용하여 2 shards 구성하는 예제.

replicaSet x 2 : arbiter + primary + secondary

config x 3

mongos x 5

아래 그림과 같은 형태.

0. 준비

다운로드

MongoDB 바이너리 파일 다운로드: http://www.mongodb.org/downloads

각 서버별로 /db/mongodb 디렉토리에 압축을 해제했다고 가정함.

경로

아래와 같다고 가정함. 없는 디렉토리는 만들 것...;

/db/mongodb/bin : 바이너리 실행파일 위치

/db/mongodb/conf : 설정파일 위치

/db/mongodb/data : 데이터 디렉토리

/db/mongodb/key : 인증을 위한 key 파일 디렉토리

/db/mongodb/logs : 로그파일 디렉토리

key 파일

인증을 위한 key파일 생성.

http://docs.mongodb.org/manual/tutorial/generate-key-file/

// key 파일 생성

[root@localhost]# openssl rand -base64 741 > /db/mongodb/key/mongodb.key

// 퍼미션 설정

[root@localhost]# chmod 0400 /db/mongodb/key/mongodb.key

// 전체 클러스터에 동일한 키 파일을 동일한 위치에 복사

1. ReplicaSet 구성

arbiter + primary + secondary 로 구성된 ReplicaSet x 2

설정파일

shard001_arbiter.conf

위치: m001.example.com /db/mongodb/conf/shard001_arbiter.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/shard001_arbiter.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/shard001_arbiter.pid

    fork: true

net:

    port: 27020

security:

    keyFile: /db/mongodb/key/mongodb.key

storage:

    dbPath: /db/mongodb/data/shard001_arbiter

    journal:

        enabled: false

    smallFiles: true

replication:

    replSetName: shard001

sharding: 

    clusterRole: shardsvr

데이터 디렉토리인 /db/mongodb/data/shard001_arbiter 생성해야 함.

shard001_primary.conf 

위치: m002.example.com /db/mongodb/conf/shard001_primary.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/shard001_primary.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/shard001_primary.pid

    fork: true

net:

    port: 27020

security:

    keyFile: /db/mongodb/key/mongodb.key

storage:

    dbPath: /db/mongodb/data/shard001_primary

replication:

    replSetName: shard001

sharding:

    clusterRole: shardsvr

데이터 디렉토리인 /db/mongodb/data/shard001_primary 생성해야 함

shard001_secondary.conf

위치: m003.example.com /db/mongodb/conf/shard001_secondary.conf

systemLog:

    path: /db/mongodb/logs/shard001_secondary.log

    logAppend: true

    destination: file

processManagement:

    pidFilePath: /db/mongodb/logs/shard001_secondary.pid

    fork: true

net:

    port: 27020

security:

    keyFile: /db/mongodb/key/mongodb.key

storage:

    dbPath: /db/mongodb/data/shard001_secondary

replication:

    replSetName: shard001

sharding:

    clusterRole: shardsvr

데이터 디렉토리인 /db/mongodb/data/shard001_secondary 생성해야 함.

프로세스 실행

각 서버에서 아래와 같이 arbiter, primary, secondary mongod를 실행해야 함.

// 각 프로세스가 실행되어야 하는 서버에서 각각 실행되어야 함.

// shard001_arbiter

[root@m001]# /db/mongodb/bin/mongod -f /db/mongodb/conf/shard001_arbiter.conf

// shard001_primary

[root@m002]# /db/mongodb/bin/mongod -f /db/mongodb/conf/shard001_primary.conf

// shard001_secondary

[root@m003]# /db/mongodb/bin/mongod -f /db/mongodb/conf/shard001_secondary.conf

실행 후 로그파일을 보면서 preallocation이 끝날 때 까지 기다려야 함.

ReplicaSet 초기화

m002 서버에서 shard001_primary용 mongod에 접속하여 ReplicaSet 구성

[root@m002]# /db/mongodb/bin/mongo localhost:27020

// ReplicaSet 초기화

> rs.initiate()

{

        "info2" : "no configuration explicitly specified -- making one",

        "me" : "m002.example.com:27020",

        "info" : "Config now saved locally.  Should come online in about a minute.",

        "ok" : 1

}

// root 추가

// ReplicaSet의 사용자는 추후 Sharding 구성 후 mongos에서 추가된 사용자와는 별개임에 주의할 것.

shard001:PRIMARY> use admin

switched to db admin

shard001:PRIMARY> db.createUser({user: "root", pwd: "비밀번호", roles: ["root"]})

Successfully added user: { "user" : "root", "roles" : [ "root" ] }

// 추가된 사용자로 로그인

shard001:PRIMARY> db.auth("root", "비밀번호")

1

// arbiter 추가

shard001:PRIMARY> rs.addArb("m001.example.com:27020")

{ "ok" : 1 }

// secondary 추가

shard001:PRIMARY> rs.add("m003.example.com:27020")

{ "ok" : 1 }

// 좀 기다렸다가 rs.status()로 제대로 추가되었는지 확인.

shard001:PRIMARY> rs.status()

{

        "set" : "shard001",

        "date" : ISODate("2014-12-29T01:32:15Z"),

        "myState" : 1,

        "members" : [

                {

                        "_id" : 0,

                        "name" : "m002.example.com:27020",

                        "health" : 1,

                        "state" : 1,

                        "stateStr" : "PRIMARY",

                        "uptime" : 184,

                        "optime" : Timestamp(1419816708, 1),

                        "optimeDate" : ISODate("2014-12-29T01:31:48Z"),

                        "electionTime" : Timestamp(1419816631, 2),

                        "electionDate" : ISODate("2014-12-29T01:30:31Z"),

                        "self" : true

                },

                {

                        "_id" : 1,

                        "name" : "m001.example.com:27020",

                        "health" : 1,

                        "state" : 7,

                        "stateStr" : "ARBITER",

                        "uptime" : 34,

                        "lastHeartbeat" : ISODate("2014-12-29T01:32:13Z"),

                        "lastHeartbeatRecv" : ISODate("2014-12-29T01:32:13Z"),

                        "pingMs" : 1

                },

                {

                        "_id" : 2,

                        "name" : "m003.example.com:27020",

                        "health" : 1,

                        "state" : 2,

                        "stateStr" : "SECONDARY",

                        "uptime" : 27,

                        "optime" : Timestamp(1419816708, 1),

                        "optimeDate" : ISODate("2014-12-29T01:31:48Z"),

                        "lastHeartbeat" : ISODate("2014-12-29T01:32:14Z"),

                        "lastHeartbeatRecv" : ISODate("2014-12-29T01:32:13Z"),

                        "pingMs" : 1,

                        "syncingTo" : "m002.example.com:27020"

                }

        ],

        "ok" : 1

}

2번째 ReplicaSet 구성 (shard002)

설정파일

shard002_arbiter.conf

위치: m001.example.com /db/mongodb/conf/shard002_arbiter.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/shard002arbiter.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/shard002_arbiter.pid

    fork: true

net:

    port: 27021

security:

    keyFile: /db/mongodb/key/mongodb.key

storage:

    dbPath: /db/mongodb/data/shard002_arbiter

    journal:

        enabled: false

    smallFiles: true

replication:

    replSetName: shard002

sharding: 

    clusterRole: shardsvr

데이터 디렉토리인 /db/mongodb/data/shard002_arbiter 생성해야 함.

shard002_primary.conf 

위치: m004.example.com /db/mongodb/conf/shard002_primary.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/shard002_primary.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/shard002_primary.pid

    fork: true

net:

    port: 27021

security:

    keyFile: /db/mongodb/key/mongodb.key

storage:

    dbPath: /db/mongodb/data/shard002_primary

replication:

    replSetName: shard002

sharding:

    clusterRole: shardsvr

데이터 디렉토리인 /db/mongodb/data/shard002_primary 생성해야 함

shard002_secondary.conf

위치: m005.example.com /db/mongodb/conf/shard002_secondary.conf

systemLog:

    path: /db/mongodb/logs/shard002_secondary.log

    logAppend: true

    destination: file

processManagement:

    pidFilePath: /db/mongodb/logs/shard002_secondary.pid

    fork: true

net:

    port: 27021

security:

    keyFile: /db/mongodb/key/mongodb.key

storage:

    dbPath: /db/mongodb/data/shard002_secondary

replication:

    replSetName: shard002

sharding:

    clusterRole: shardsvr

데이터 디렉토리인 /db/mongodb/data/shard002_secondary 생성해야 함.

프로세스 실행

각 서버에서 아래와 같이 arbiter, primary, secondary mongod를 실행해야 함.

// 각 프로세스가 실행되어야 하는 서버에서 각각 실행되어야 함.

// shard002_arbiter

[root@m001]# /db/mongodb/bin/mongod -f /db/mongodb/conf/shard002_arbiter.conf

// shard002_primary

[root@m004]# /db/mongodb/bin/mongod -f /db/mongodb/conf/shard002_primary.conf

// shard002_secondary

[root@m005]# /db/mongodb/bin/mongod -f /db/mongodb/conf/shard002_secondary.conf

실행 후 로그파일을 보면서 preallocation이 끝날 때 까지 기다려야 함.

ReplicaSet 초기화

m004 서버에서 shard002_primary용 mongod에 접속하여 ReplicaSet 구성

[root@m004]# /db/mongodb/bin/mongo localhost:27021

// ReplicaSet 초기화

> rs.initiate()

{

        "info2" : "no configuration explicitly specified -- making one",

        "me" : "m004.example.com:27021",

        "info" : "Config now saved locally.  Should come online in about a minute.",

        "ok" : 1

}

// root 추가

// ReplicaSet의 사용자는 추후 Sharding 구성 후 mongos에서 추가된 사용자와는 별개임에 주의할 것.

shard002:PRIMARY> use admin

switched to db admin

shard002:PRIMARY> db.createUser({user: "root", pwd: "비밀번호", roles: ["root"]})

Successfully added user: { "user" : "root", "roles" : [ "root" ] }

// 추가된 사용자로 로그인

shard002:PRIMARY> db.auth("root", "비밀번호")

1

// arbiter 추가

shard002:PRIMARY> rs.addArb("m001.example.com:27021")

{ "ok" : 1 }

// secondary 추가

shard002:PRIMARY> rs.add("m005.example.com:27021")

{ "ok" : 1 }

// 좀 기다렸다가 rs.status()로 제대로 추가되었는지 확인.

shard002:PRIMARY> rs.status()

{

        "set" : "shard002",

        "date" : ISODate("2014-12-29T01:32:15Z"),

        "myState" : 1,

        "members" : [

                {

                        "_id" : 0,

                        "name" : "m004.example.com:27021",

                        "health" : 1,

                        "state" : 1,

                        "stateStr" : "PRIMARY",

                        "uptime" : 184,

                        "optime" : Timestamp(1419816708, 1),

                        "optimeDate" : ISODate("2014-12-29T01:31:48Z"),

                        "electionTime" : Timestamp(1419816631, 2),

                        "electionDate" : ISODate("2014-12-29T01:30:31Z"),

                        "self" : true

                },

                {

                        "_id" : 1,

                        "name" : "m001.example.com:27021",

                        "health" : 1,

                        "state" : 7,

                        "stateStr" : "ARBITER",

                        "uptime" : 34,

                        "lastHeartbeat" : ISODate("2014-12-29T01:32:13Z"),

                        "lastHeartbeatRecv" : ISODate("2014-12-29T01:32:13Z"),

                        "pingMs" : 1

                },

                {

                        "_id" : 2,

                        "name" : "m005.example.com:27021",

                        "health" : 1,

                        "state" : 2,

                        "stateStr" : "SECONDARY",

                        "uptime" : 27,

                        "optime" : Timestamp(1419816708, 1),

                        "optimeDate" : ISODate("2014-12-29T01:31:48Z"),

                        "lastHeartbeat" : ISODate("2014-12-29T01:32:14Z"),

                        "lastHeartbeatRecv" : ISODate("2014-12-29T01:32:13Z"),

                        "pingMs" : 1,

                        "syncingTo" : "m004.example.com:27021"

                }

        ],

        "ok" : 1

}

여기까지 하면 sharding을 구성할 ReplicaSet 2개 준비 완료.

2. config

sharding cluster의 meta data를 저장할 config 서버 구성

설정파일

config001.conf

위치: m001.example.com /db/mongodb/conf/config001.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/config001.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/config001.pid

    fork: true

net:

    port: 27019

security:

    keyFile: /db/mongodb/key/mongodb.key

storage:

    dbPath: /db/mongodb/data/config001

sharding:

    clusterRole: configsvr  

데이터 디렉토리인 /db/mongodb/data/config001 생성해야 함.

config002.conf

위치: m002.example.com /db/mongodb/conf/config002.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/config002.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/config002.pid

    fork: true

net:

    port: 27019

security:

    keyFile: /db/mongodb/key/mongodb.key

storage:

    dbPath: /db/mongodb/data/config002

sharding:

    clusterRole: configsvr 

데이터 디렉토리인 /db/mongodb/data/config002 생성해야 함.

config003.conf

위치: m004.example.com /db/mongodb/conf/config003.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/config003.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/config003.pid

    fork: true

net:

    port: 27019

security:

    keyFile: /db/mongodb/key/mongodb.key

storage:

    dbPath: /db/mongodb/data/config003

sharding:

    clusterRole: configsvr 

데이터 디렉토리인 /db/mongodb/data/config003 생성해야 함.

프로세스 실행

각 서버에서 아래와 같이 config용 mongod를 실행해야 함.

// 각 프로세스가 실행되어야 하는 서버에서 각각 실행되어야 함.

// config001

[root@m001]# /db/mongodb/bin/mongod -f /db/mongodb/conf/config001.conf

// config002

[root@m002]# /db/mongodb/bin/mongod -f /db/mongodb/conf/config002.conf

// config003

[root@m004]# /db/mongodb/bin/mongod -f /db/mongodb/conf/config003.conf

로그를 보면서 config 서버들이 전부 올라올 때까지 대기

3. mongos

MongoDB cluster와 외부를 이어주는 router인 mongos 구성.

설정파일

mongos001.conf

위치: m001.example.com /db/mongodb/conf/mongos001.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/mongos001.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/mongos001.pid

    fork: true

net:

    port: 27017

security:

    keyFile: /db/mongodb/key/mongodb.key

sharding:

    configDB: m001.example.com:27019,m002.example.com:27019,m004.example.com:27019 

mongos002.conf

위치: m002.example.com /db/mongodb/conf/mongos002.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/mongos002.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/mongos002.pid

    fork: true

net:

    port: 27017

security:

    keyFile: /db/mongodb/key/mongodb.key

sharding:

    configDB: m001.example.com:27019,m002.example.com:27019,m004.example.com:27019 

mongos003.conf

위치: m003.example.com /db/mongodb/conf/mongos003.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/mongos003.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/mongos003.pid

    fork: true

net:

    port: 27017

security:

    keyFile: /db/mongodb/key/mongodb.key

sharding:

    configDB: m001.example.com:27019,m002.example.com:27019,m004.example.com:27019 

mongos004.conf

위치: m004.example.com /db/mongodb/conf/mongos004.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/mongos004.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/mongos004.pid

    fork: true

net:

    port: 27017

security:

    keyFile: /db/mongodb/key/mongodb.key

sharding:

    configDB: m001.example.com:27019,m002.example.com:27019,m004.example.com:27019 

mongos005.conf

위치: m001.example.com /db/mongodb/conf/mongos001.conf

systemLog:

    destination: file

    path: /db/mongodb/logs/mongos005.log

    logAppend: true

processManagement:

    pidFilePath: /db/mongodb/logs/mongos005.pid

    fork: true

net:

    port: 27017

security:

    keyFile: /db/mongodb/key/mongodb.key

sharding:

    configDB: m001.example.com:27019,m002.example.com:27019,m004.example.com:27019 

프로세스 실행

각 서버에서 아래와 같이 mongos 프로세스를 실행해야 함.

// 각 프로세스가 실행되어야 하는 서버에서 각각 실행되어야 함.

// mongos001

[root@m001]# /db/mongodb/bin/mongos -f /db/mongodb/conf/mongos001.conf

// mongos002

[root@m002]# /db/mongodb/bin/mongos -f /db/mongodb/conf/mongos002.conf

// mongos003

[root@m003]# /db/mongodb/bin/mongos -f /db/mongodb/conf/mongos003.conf

// mongos004

[root@m004]# /db/mongodb/bin/mongos -f /db/mongodb/conf/mongos004.conf

// mongos005

[root@m005]# /db/mongodb/bin/mongos -f /db/mongodb/conf/mongos005.conf

4. ReplicaSet을 Shard로 추가

// mongos 접속

[root@m001]# /db/mongodb/bin/mongo localhost:27017

// shard 추가

mongos> sh.addShard("shard001/m001.example.com:27020,m002.example.com:27020,m003.example.com:27020")

{ "shardAdded" : "shard001", "ok" : 1 }

mongos> sh.addShard("shard002/m001.example.com:27021,m004.example.com:27021,m005.example.com:27021")

{ "shardAdded" : "shard002", "ok" : 1 }

// 확인

mongos> use config

switched to db config

mongos> db.shards.find()

{ "_id" : "shard001", "host" : "shard001/m002.example.com:27020,m003.example.com:27020" }

{ "_id" : "shard002", "host" : "shard002/m004.example.com:27021,m005.example.com:27021" }

5. root 추가

mongos> use admin

switched to db admin

mongos> db.createUser({user: "root", pwd: "비밀번호", roles: ["root"]})

Successfully added user: { "user" : "root", "roles" : [ "root" ] }

// 확인

mongos> db.auth("root", "비밀번호")

1

6. Sharding

// sharding할 DB 선택

mongos> use mydb

switched to db mydb

// DB sharding

mongos> sh.enableSharding("mydb")

{ "ok" : 1 }

// shard key로 사용할 인덱스 생성

mongos> db.mycol.ensureIndex({idx1: 1, idx2: 1})

{

        "raw" : {

                "shard001/m002.example.com:27020,m003.example.com:27020" : {

                        "createdCollectionAutomatically" : true,

                        "numIndexesBefore" : 1,

                        "numIndexesAfter" : 2,

                        "ok" : 1

                }

        },

        "ok" : 1

}

// 생성한 인덱스를 이용하여 collection sharding

mongos> sh.shardCollection("mydb.mycol", {idx1: 1, idx2: 1})

{ "collectionsharded" : "mydb.mycol", "ok" : 1 }