twemproxy,也叫nutcraker。是一个twtter开源的一个redis和memcache代理服务器。 redis作为一个高效的缓存服务器,非常具有应用价值。但是当使用比较多的时候,就希望可以通过某种方式 统一进行管理。避免每个应用每个客户端管理连接的松散性。同时在一定程度上变得可以控制。
Twemproxy是一种代理分片机制,由Twitter开源。Twemproxy作为代理,可接受来自多个程序的访问,按照路由规则,转发给后台的各个Redis服务器,再原路返回。该方案很好的解决了单个Redis实例承载能力的问题。当然,Twemproxy本身也是单点,需要用Keepalived做高可用方案。通过Twemproxy可以使用多台服务器来水平扩张redis服务,可以有效的避免单点故障问题。虽然使用Twemproxy需要更多的硬件资源和在redis性能有一定的损失(twitter测试约20%),但是能够提高整个系统的HA也是相当划算的。不熟悉twemproxy的同学,如果玩过nginx反向代理或者mysql proxy,那么你肯定也懂twemproxy了。其实twemproxy不光实现了redis协议,还实现了memcached协议,什么意思?换句话说,twemproxy不光可以代理redis,还可以代理memcached,官方说明:
twemproxy (pronounced "two-em-proxy"), aka nutcracker is a fast and lightweight proxy for memcachedand redis protocol. It was built primarily to reduce the number of connections to the caching servers on the backend. This, together with protocol pipeling and sharding enables you to horizontally scale your distributed caching architecture.
Features
Fast.
Lightweight.
Maintains persistent server connections.
Keeps connection count on the backend caching servers low.
Enables pipelining of requests and responses.
Supports proxying to multiple servers.
Supports multiple server pools simultaneously.
Shard data automatically across multiple servers.
Implements the complete memcached ascii and redis protocol.
Easy configuration of server pools through a YAML file.
Supports multiple hashing modes including consistent hashing and distribution.
Can be configured to disable nodes on failures.
Observability via stats exposed on the stats monitoring port.
Works with Linux, *BSD, OS X and SmartOS (Solaris)
twemproxy
-
支持失败节点自动删除
- 可以设置重新连接该节点的时间
- 可以设置连接多少次之后删除该节点
- 该方式适合作为cache存储
-
支持设置HashTag
- 通过HashTag可以自己设定将两个KEYhash到同一个实例上去。
-
减少与redis的直接连接数
- 保持与redis的长连接
- 可设置代理与后台每个redis连接的数目
-
自动分片到后端多个redis实例上
- 多种hash算法(部分还没有研究明白)
- 可以设置后端实例的权重
-
避免单点问题
- 可以平行部署多个代理层.client自动选择可用的一个
-
支持redis pipelining request
-
支持状态监控
- 可设置状态监控ip和端口,访问ip和端口可以得到一个json格式的状态信息串
- 可设置监控信息刷新间隔时间
-
高吞吐量
- 连接复用,内存复用。
- 将多个连接请求,组成reids pipelining统一向redis请求。
Zero Copy
In twemproxy, all the memory for incoming requests and outgoing responses is allocated in mbuf. Mbuf enables zero-copy because the same buffer on which a request was received from the client is used for forwarding it to the server. Similarly the same mbuf on which a response was received from the server is used for forwarding it to the client.
Furthermore, memory for mbufs is managed using a reuse pool. This means that once mbuf is allocated, it is not deallocated, but just put back into the reuse pool. By default each mbuf chunk is set to 16K bytes in size. There is a trade-off between the mbuf size and number of concurrent connections twemproxy can support. A large mbuf size reduces the number of read syscalls made by twemproxy when reading requests or responses. However, with a large mbuf size, every active connection would use up 16K bytes of buffer which might be an issue when twemproxy is handling large number of concurrent connections from clients. When twemproxy is meant to handle a large number of concurrent client connections, you should set chunk size to a small value like 512 bytes using the -m or --mbuf-size=N argument.
Configuration
Twemproxy can be configured through a YAML file specified by the -c or --conf-file command-line argument on process start. The configuration file is used to specify the server pools and the servers within each pool that twemproxy manages. The configuration files parses and understands the following keys:
- listen: The listening address and port (name:port or ip:port) or an absolute path to sock file (e.g. /var/run/nutcracker.sock) for this server pool.
- client_connections: The maximum number of connections allowed from redis clients. Unlimited by default, though OS-imposed limitations will still apply.
- hash: The name of the hash function. Possible values are:
- one_at_a_time
- md5
- crc16
- crc32 (crc32 implementation compatible with libmemcached)
- crc32a (correct crc32 implementation as per the spec)
- fnv1_64
- fnv1a_64
- fnv1_32
- fnv1a_32
- hsieh
- murmur
- jenkins
- hash_tag: A two character string that specifies the part of the key used for hashing. Eg "{}" or "$$". Hash tag enable mapping different keys to the same server as long as the part of the key within the tag is the same.
- distribution: The key distribution mode. Possible values are:
- ketama
- modula
- random
- timeout: The timeout value in msec that we wait for to establish a connection to the server or receive a response from a server. By default, we wait indefinitely.
- backlog: The TCP backlog argument. Defaults to 512.
- preconnect: A boolean value that controls if twemproxy should preconnect to all the servers in this pool on process start. Defaults to false.
- redis: A boolean value that controls if a server pool speaks redis or memcached protocol. Defaults to false.
- redis_auth: Authenticate to the Redis server on connect.
- redis_db: The DB number to use on the pool servers. Defaults to 0. Note: Twemproxy will always present itself to clients as DB 0.
- server_connections: The maximum number of connections that can be opened to each server. By default, we open at most 1 server connection.
- auto_eject_hosts: A boolean value that controls if server should be ejected temporarily when it fails consecutively server_failure_limit times. See liveness recommendations for information. Defaults to false.
- server_retry_timeout: The timeout value in msec to wait for before retrying on a temporarily ejected server, when auto_eject_host is set to true. Defaults to 30000 msec.
- server_failure_limit: The number of consecutive failures on a server that would lead to it being temporarily ejected when auto_eject_host is set to true. Defaults to 2.
- servers: A list of server address, port and weight (name:port:weight or ip:port:weight) for this server pool.
相关推荐
Twemproxy 构建redis集群 twemproxy最新版本 0.4.1
twemproxy源码包
多线程版的Twemproxies,线性提高Twemproxy的处理能力,使Twemproxy不再成为整个集群中的短板
redis集群代理Twemproxy 安装过程,Twemproxy是一个专门为了这种nosql数据库设计的一款代理工具软件,这个工具软件最大的特征是可以实现数据的分片处理。所谓的分片指的是根据一定的算法将要保存的数据保存到不同的...
Laravel开发-laravel-twemproxy 一个图书馆帮助老拉雷维尔与两个代理工作
twemproxy,也叫nutcraker。是一个twtter开源的一个redis和memcache代理服务器。 redis作为一个高效的缓存服务器,非常具有应用价值。但是当使用比较多的时候,就希望可以通过某种方式 统一进行管理。避免每个应用每...
描述安装/配置 twemproxy要求平台: Ubuntu食谱: 易于属性node['twemproxy']['install_method'] - 默认为"package" 。 node['twemproxy']['repo'] - 默认为"ppa"可以为零。 node['twemproxy']['config_file'] - ...
keepalive+twemproxy+ redis主从安装配置的安装文档,里头包括了6篇
一个由twitter开源的c版本proxy,同时支持memcached和redis=。此版本为可以configure安装的版本
Redis 存储分片之代理服务Twemproxy 测试.pdfRedis 存储分片之代理服务Twemproxy 测试.pdf
美团twemproxy是memcached和redis协议的多进程,快速和轻量级代理
本次修改用于支持twemproxy支持ssdb协议,支持ssdb server的读写分离和分片支持
twemproxy:这是一个 redis 代理,其主要功能就是在多个 redis 节点和访问者之间做路由分发:访问者不需要自己判断某个 key 该存储到哪一个 redis 节点或者该到哪个节点去获取某个 key 的值——这是一个负载均衡组件
属性default.rb 属性类型描述选项默认['twemproxy'] ['mirror'] 细绳神器镜-- ['twemproxy'] ['version'] 细绳神器版本-- v0.4.1 ['twemproxy'] ['file_name'] 细绳工件名称默认为nil,因为它是从github档案中获取的...
Twemproxy需要的依赖,加Redis安装包,tcl-8.5.7-6.el6.x86_64
Twemproxy通过引入一个代理层,可以将其后端的多台Redis或Memcached实例进行统一管理与分配,使应用程序只需要在Twemproxy上进行操作,而不用关心后面具体有多少个真实的Redis或Memcached存储,有了Twemproxy,...
twemproxy高山Docker映像构建 建造 您需要docker 17.05才能构建 docker build -t yourname/twemproxy:0.4.1 . 跑步 通过ENV SERVERS以逗号分隔的服务器列表 docker run -e SERVERS=ip1:6379:1,ip2:6379:1 -p 6379:...
本安装包围安装twemproxy时,必备源码包,配套autoconf2.69。 助你顺逆安装一臂之力。
twemproxy(胡桃夹子) twemproxy (发音为“ two-em-proxy”),又名胡桃夹子,是一种用于和协议的快速轻量级代理。 它的建立主要是为了减少与后端缓存服务器的连接数。 这与协议流水线和分片一起使您能够水平扩展...
NULL 博文链接:https://jinnianshilongnian.iteye.com/blog/2186316