MongoDB auto shard配置说明 -

landyer

浏览: 138708 次
性别:
来自: 上海

最近访客更多访客>>

bill00

hubobocbb

ronggui

笨鸟刃心

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

MongoDB auto shard配置说明

博客分类：

mongodb

MongoDB Linux Git Web Blog

mongodb支持自动分片功能，这无疑给DBA和开发人员带来很大便利，我们无需要手动按一定规则进行拆分DB，当数据量增大后，我们只需要扩充设备即可。

mongodb shard配置文档如下：

auo shard 试验设备列表

主机列表

主机名 ip 用途监听端口 web访问url

localhost 192.168.100.245 config server 27019 http://192.168.100.245:28019/

TEST40 192.168.100.212 shard 1 27018 http://192.168.100.212:28018/

chn-xg-5-5c1 192.168.100.210 shard 2 27018 http://192.168.100.210:28018/

band-all 192.168.100.229 mongos process/mongos Router 27017 http://192.168.100.229:27017/

分析发现config server、shard、mongos的监听端口和web端口均是第二位的差异。监听端口第二位是7，web端口是8，其它全部相同。

mongod --fork --logpath /var/log/mongodb.log --logappend

mogod默认启动为前台进程，当连接session断开，mongod程序自动关闭

一、配置shard 各个模块

1、启动shard1

[root@TEST40 ~]# mongod --shardsvr --fork --logpath /var/log/mongodb.log --logappend

[1] 28521

[root@TEST40 ~]# Tue Aug 17 03:01:57 MongoDB starting : pid=28521 port=27018 dbpath=/data/db/ 32-bit

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data

** see http://blog.mongodb.org/post/137788967/32-bit-limitations

Tue Aug 17 03:01:57 db version v1.6.0, pdfile version 4.5

Tue Aug 17 03:01:57 git version: 2c7f164b653f0d703947572ede064aed41cc2185

Tue Aug 17 03:01:57 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37

Tue Aug 17 03:01:57 [initandlisten] waiting for connections on port 27018

Tue Aug 17 03:01:57 [websvr] web admin interface listening on port 28018

2、启动shard2，shard2的dbpath与默认路径不同，因此需要制定dbpath

[root@chn-xg-5-5c1 cache2]# mongod --shardsvr --dbpath /data/cache2/db --fork --logpath /var/log/mongodb.log --logappend

[1] 31264

[root@chn-xg-5-5c1 cache2]# Tue Aug 17 03:11:48 MongoDB starting : pid=31264 port=27018 dbpath=/data/cache2/db 32-bit

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data

** see http://blog.mongodb.org/post/137788967/32-bit-limitations

Tue Aug 17 03:11:48 db version v1.6.0, pdfile version 4.5

Tue Aug 17 03:11:48 git version: 2c7f164b653f0d703947572ede064aed41cc2185

Tue Aug 17 03:11:48 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37

Tue Aug 17 03:11:48 [initandlisten] waiting for connections on port 27018

Tue Aug 17 03:11:48 [websvr] web admin interface listening on port 28018

[root@chn-xg-5-5c1 cache2]#

3、配置config server，config server需要dbpath和repairpath

dbpath默认为/data/configdb，repairpath默认为/data/db/，这与shard是不同的

[root@localhost data]# mongod --configsvr --fork --logpath /var/log/mongodb.log --logappend

[1] 2144

[root@localhost data]# Tue Aug 17 03:13:54 MongoDB starting : pid=2144 port=27019 dbpath=/data/configdb 32-bit

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data

** see http://blog.mongodb.org/post/137788967/32-bit-limitations

Tue Aug 17 03:13:54 db version v1.6.0, pdfile version 4.5

Tue Aug 17 03:13:54 git version: 2c7f164b653f0d703947572ede064aed41cc2185

Tue Aug 17 03:13:54 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37

Tue Aug 17 03:13:54 [initandlisten] diagLogging = 1

Tue Aug 17 03:13:54 [initandlisten] waiting for connections on port 27019

Tue Aug 17 03:13:54 [websvr] web admin interface listening on port 28019

[root@localhost data]#

4、配置mongos router

[root@band-all local]# mongos --configdb 192.168.100.245:27019 --fork --logpath /var/log/mongodb.log --logappend

[1] 6384

[root@band-all local]# Tue Aug 17 03:46:12 mongos db version v1.6.0, pdfile version 4.5 starting (--help for usage)

Tue Aug 17 03:46:12 git version: 2c7f164b653f0d703947572ede064aed41cc2185

Tue Aug 17 03:46:12 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37

Tue Aug 17 03:46:12 [websvr] web admin interface listening on port 28017

Tue Aug 17 03:46:12 waiting for connections on port 27017

[root@band-all local]#

二、配置shard 集群

1、登陆mongos server，添加shard

[root@localhost data]# mongo 192.168.100.229:27017/admin

MongoDB shell version: 1.6.0

connecting to: 192.168.100.229:27017/admin

Tue Aug 17 05:59:38 [conn4] building new index on { _id: 1 } for config.databases

Tue Aug 17 05:59:38 [conn4] Buildindex config.databases idxNo:0 { name: "_id_", ns: "config.databases", key: { _id: 1 } }

Tue Aug 17 05:59:38 [conn4] done for 0 records 0.025secs

添加shard 1

> db.runCommand( { addshard : "192.168.100.210:27018" } );

{ "shardAdded" : "shard0000", "ok" : 1 }

添加shard 2

创建shard时，两个shard不能有相同的数据库

> db.runCommand( { addshard : "192.168.100.212:27018" } );

{

"ok" : 0,

"errmsg" : "trying to add shard 192.168.100.212:27018 because local database recommender exists in another shard0000:192.168.100.210:27018"

}

可以在192.168.100.212:27018通过 db.dropDatabase() 删除recommender

db.runCommand( { addshard : "192.168.100.212:27018" } );

查看shard节点列表

> db.runCommand( { listshards : 1 } );

{

"shards" : [

{

"_id" : "shard0000",

"host" : "192.168.100.210:27018"

{

"_id" : "shard0001",

"host" : "192.168.100.212:27018"

}

"ok" : 1

}

将数据库设置为shard模式

> db.runCommand( { enablesharding : "test" } );

{ "ok" : 1 }

> db.runCommand( { enablesharding : "recommender" } );

{ "ok" : 1 }

2、登陆mongos查看对集合进行分片

[root@band-all data]# mongo 192.168.100.229:27017/admin

MongoDB shell version: 1.6.0

connecting to: 192.168.100.229:27017/admin

> show dbs

admin

local

> use admin

switched to db admin

> db.runCommand({listshards:1})

{

"shards" : [

{

"_id" : "shard0000",

"host" : "192.168.100.210:27018"

{

"_id" : "shard0001",

"host" : "192.168.100.212:27018"

}

"ok" : 1

}

> db.printShardingStatus();

--- Sharding Status ---

sharding version: { "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard0000", "host" : "192.168.100.210:27018" }

{ "_id" : "shard0001", "host" : "192.168.100.212:27018" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" }

> config = connect("192.168.100.245:27019") //登陆config server

connecting to: 192.168.100.245:27019

192.168.100.245:27019

> test = db.getSisterDB("test") //新增数据库test

> db.runCommand( { enablesharding : "test" } ) //将test 数据库进行分片

> db.runCommand( { shardcollection : "test.people", key : {name : 1} } ) //将数据集people进行分片

> db.printShardingStatus();

--- Sharding Status ---

sharding version: { "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard0000", "host" : "192.168.100.210:27018" }

{ "_id" : "shard0001", "host" : "192.168.100.212:27018" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" }

{ "_id" : "test", "partitioned" : true, "primary" : "shard0000" }

test.people chunks:

{ "name" : { $minKey : 1 } } -->> { "name" : { $maxKey : 1 } } on : shard0000 { "t" : 1000, "i" : 0 }

> use test

switched to db test

> show collections

people

system.indexes

批量插入数据

for (var i = 1; i <= 200000; i++) db.people.save({name:"fly_people",sex:"female",age:25});

for (var i = 1; i <= 20000000; i++) db.people.save({name:"houxuefeng",sex:"male",age:30});

> db.people.stats()

{

"sharded" : true,

"ns" : "test.people",

"count" : 462008,

"size" : 33264600,

"avgObjSize" : 72.00005194715243,

"storageSize" : 902522624,

"nindexes" : 2,

"nchunks" : 4,

"shards" : {

"shard0000" : {

"ns" : "test.people",

"count" : 402006,

"size" : 28944448,

"avgObjSize" : 72.00003980040098,

"storageSize" : 891340544,

"numExtents" : 22,

"nindexes" : 2,

"lastExtentSize" : 155079936,

"paddingFactor" : 1,

"flags" : 1,

"totalIndexSize" : 58966016,

"indexSizes" : {

"_id_" : 18694144,

"name_1" : 40271872

"ok" : 1

"shard0001" : {

"ns" : "test.people",

"count" : 60002,

"size" : 4320152,

"avgObjSize" : 72.00013332888904,

"storageSize" : 11182080,

"numExtents" : 6,

"nindexes" : 2,

"lastExtentSize" : 8388608,

"paddingFactor" : 1,

"flags" : 1,

"totalIndexSize" : 5275648,

"indexSizes" : {

"_id_" : 2498560,

"name_1" : 2777088

"ok" : 1

}

"ok" : 1

}

至此MongoDB auto shard配置完毕

备注：

问题1、

当test.people数据量达到一定量时，这也就是32位限制，新数据无法入库，查看数据库文件也不在增加

> for (var i = 1; i <= 20000; i++) db.people.save({name:"houxuefeng",sex:"male",age:30});

can't map file memory - mongo requires 64 bit build for larger datasets

> db.people.stats();

{

"ns" : "test.people",

"count" : 10128817,

"size" : 729274896,

"avgObjSize" : 72.00000710843132,

"storageSize" : 891340544,

"numExtents" : 22,

"nindexes" : 2,

"lastExtentSize" : 155079936,

"paddingFactor" : 1,

"flags" : 1,

"totalIndexSize" : 1241788192,

"indexSizes" : {

"_id_" : 420325264,

"name_1" : 821462928

"ok" : 1

}

[root@chn-xg-5-5c1 db]# ll -lh

总用量 3.0G

-rwxr-xr-x 1 root root 6 8月 18 17:42 mongod.lock

drwxr-xr-x 3 root root 4.0K 8月 19 10:48 moveChunk

-rw------- 1 root root 64M 8月 18 17:56 test.0

-rw------- 1 root root 128M 8月 18 17:56 test.1

-rw------- 1 root root 256M 8月 19 10:46 test.2

-rw------- 1 root root 512M 8月 19 10:48 test.3

-rw------- 1 root root 512M 8月 19 10:52 test.4

-rw------- 1 root root 512M 8月 19 10:58 test.5

-rw------- 1 root root 512M 8月 19 11:04 test.6

-rw------- 1 root root 512M 8月 19 11:12 test.7

-rw------- 1 root root 16M 8月 18 17:56 test.ns

drwxr-xr-x 2 root root 4.0K 8月 18 17:58 _tmp

[root@chn-xg-5-5c1 db]#

问题2、

创建shard时，两个shard不能有相同的数据库

> db.runCommand( { addshard : "192.168.100.212:27018" } );

{

"ok" : 0,

"errmsg" : "trying to add shard 192.168.100.212:27018 because local database recommender exists in another shard0000:192.168.100.210:27018"

}

可以在192.168.100.212:27018通过 db.dropDatabase() 删除recommender

问题3、

replica set 配置完毕后才可以添加为shard，否则会出错

> db.runCommand( { addshard : "foo/192.168.100.212:27018,192.168.100.212:27019,192.168.100.212:27020" } );

{ "shardAdded" : "shard0001", "ok" : 1 }

分享到：

基于Mongodb进行分布式数据存储方法四 | mongoDB 入门指南、命令行示例

2011-06-10 16:50
浏览 2299
评论(0)
分类:数据库
查看更多

发表评论

您还没有登录,请您登录后再发表评论

最近访客更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论

MongoDB auto shard配置说明

评论

发表评论

相关推荐

最近访客 更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论

MongoDB auto shard配置说明

评论

发表评论

相关推荐

mongo安装命令笔记

mongodb命令大全

MongoDB1.8安装、分布式自动分片(Auto-Sharding)配置备忘

基于Mongodb进行分布式数据存储 方法四

mongoDB 入门指南、命令行示例

PHP操作MongoDB数据库附数据库安装

配置mongodb分片群集(sharding cluster) 方法三

ubuntu MonogoDB Sharding + Replica Set配置集群 方法二

搭建MongoDB Sharding集群 方法一

最近访客更多访客>>

基于Mongodb进行分布式数据存储方法四

ubuntu MonogoDB Sharding + Replica Set配置集群方法二

搭建MongoDB Sharding集群方法一