http://wiki.apache.org/hama/Architecture
Components
Apache Hama, based on Bulk Synchronous Parallel model[1], comprises three major components:
- BSPMaster
- Zookeeper.
It is very similar with Hadoop architecture, only except the portion of communication and synchronization mechanisms.
In a normal usecase the user submits a so called "Job" which is a definition of how to run a computation. A job once submitted will have multiple tasks that are launched across the cluster.
BSPMaster
BSPMaster is responsible for the following:
-
Maintaining its own state.
- Maintaining groom server status.
- Maintaining supersteps and other counters in a cluster.
-
Maintaining jobs and tasks.
- Scheduling Jobs and assigning tasks to groom servers
- Distributing execution classes and configuration across groom servers.
- Providing users with the cluster control interface (web and console based).
A BSP Master and multiple grooms are started by the script. Then, the bsp master starts up with a RPC server to which groom servers can dynamically register itself. Groom servers starts up with a BSPPeer instance - later, BSPPeer needs to be integrated with GroomServer - and a RPC proxy to contact the bsp master. After started, each groom periodically sends a heartbeat message that encloses its groom server status, including maximum task capacity, unused memory, and so on.
Each time the bsp master receives a heartbeat message, it brings up-to-date groom server status - the bsp master makes use of groom servers' status in order to effectively assign tasks to idle groom servers - and returns a heartbeat response that contains assigned tasks and others actions that a groom server has to do. For now, we have a FIFO job scheduler and very simple task assignment algorithms.
GroomServer
A Groom Server (shortly referred to as groom) is a process that manages life cycle of bsp tasks assigned by BSPMaster. Each groom contacts the BSPMaster, and reports task statuses by means of periodical piggybacks with BSPMaster. Each groom is designed to run with HDFS or other distributed storages. Basically, a groom server and a data node should run on one physical node to get the best performance for data-locality. Note that in a massive parallel environment, the benefit of data locality is lost when large amount of virtual processes must be multiplexed onto physical processes[2].
Zookeeper
A Zookeeper is used to manage the efficient barrier synchronization of the BSPPeers. Later, it will also be used for the area of a fault tolerance system.
Communication and Synchronization Process
Each BSP task has a set of Outgoing Message Manager and Incoming Queue.
Outgoing Message Manager collects the message to be sent, serializes it, compresses it and puts it in a bundles. At barrier synchronization phase, each BSP task exchanges the bundles, deserializes it, decompresses it and puts it into the Incoming Queue.
System Diagram
- BSPMaster starts up
-
GroomServer starts up
-
ZooKeeper cluster starts up
-
GroomServer dynamically registers itself to BSPMaster
-
GroomServer forks/ manages BSPPeer(s)
-
BSPPeers communicate/ perform barrier synchronization through ZooKeeper cluster.
Reference
[1]. Valiant, Leslie G., A bridging model for parallel computation.
[2]. David B. Skillicorn, Jonathan M. D. Hill, and W. F. McColl. Questions and Answers about BSP. Scientific Programming, 6(3):249-274, Fall 1997.
相关推荐
分布式模式的Hama安装笔记,内容如下: 1.参考“hadoop安装.txt”,完成hadoop的安装。节点信息如下: 192.168.1.160 hadoop-1 192.168.1.161 hadoop-2 192.168.1.162 hadoop-3 2.添加环境变量 在/etc/profile...
汉密顿焦虑量表(HAMA) 填表注意事项:在最适合病人情况中划一个钩“√”,所有项目采用0~4分的5级评分法,各级的标准为:(0)为无症状;(1)轻;(2)中等;(3)重;(4)极重。 无症状 轻 中等 重 极重 1、...
1.Hama-0.6.0,里面含有安装版和源码两部分,与hadoop-1.0.3 结合使用,已在Linux(RHEL和Ubuntu)平台测试通过。 2. Hama是基于HDFS上的BSP模型实现。
基于Hama并行计算框架的多层级作业调度算法的研究及实现 胡月胜
refined.zip,scala的简单求精类型scala的简单求精类型
HAMA焦虑量表.doc
HAMA抑郁量表.pdf
2021年HAMA焦虑量表
哈马珠 Hama Beads 编辑器
汉密尔顿焦虑量表HAMA项打印版.pdf
汉密尔顿焦虑量表HAMA(14项打印版)-2页.pdf
基于Hama并行计算框架的多层级作业调度算法的研究及实现.pdf
综合护理干预对无肝素血液透析患者HAMA凝血程度及不良反应率的影响分析
Arduino库,用于从Ventus W174 / W132(已测试),Auriol H13726,Hama EWS 1500,Meteoscan W155 / W160读取天气数据 此处描述了气象站的传输协议: : 可以通过接收器模块RXB6 / MX-RM-5V进行通信,也可以直接...
赠送jar包:hppc-0.7.1.jar; 赠送原API文档:hppc-0.7.1-javadoc.jar; 赠送源代码:hppc-0.7.1-sources.jar; 赠送Maven依赖信息文件:hppc-0.7.1.pom; 包含翻译后的API文档:hppc-0.7.1-javadoc-API文档-中文...
2008年5月Hama被视为Apache众多项目中一个被孵化的项目,目前(2010年12月)在Hama的项目网站上还没有正式的release版本,作为Hadoop项目中的一个子项目,BSP模型是Hama计算的核心,并且实现了分布式的计算框架,采用...