similar to other open source projects,spark has several shells are listed there
sbin | server side shells | ||
start-all.sh | start the whole spark daemons | (ie. start-master.sh,start-slaves.sh) | |
start-master.sh | startup the spark's master process | deliver to "spark-daemon.sh start master.Master" | |
start-slaves.sh | starts all workers | deliver to sbin/slaves.sh | |
spark-daemon.sh | spawn up any daemons,e.g. spark-class.sh,spark-submit.sh.usage: spark-daemon.sh [--config <conf-dir>] (start|stop|status) <spark-command> <spark-instance-number> <args...> |
for start-master.sh,here will deliver to bin/spark-class.sh | |
slaves.sh | login into all slaves dfined in conf/slaves,then issue sbin/start-slave.sh | ||
spark-config.sh | export some global variables,e.g. SPARK_HOME | ||
start-slave.sh | deliver to spark-daemon.sh start deploy.worker.Worker |
some settings,eg. SPARK_WORKER_INSTANCES |
|
bin | |||
load-spark-env.sh | load the file spark-env.sh if exists | ||
spark-class.sh[end] |
finally,spawn up a daemon(class) returned by executing "org.apache.spark.launcher.Main",the class is specified by caller.(eg. spark-daemon.sh) Usage: spark-class <class> [<args>] |
||
spark-shell |
an interactive interface with spark to test,demo,committing spark app. Help msg is grapped from spark-submit.it uses athird-party jar named Jline to simulate a shell style. |
it will deliver to spark-submit shell.ie. bin/spark-submit --class org.apache.spark.repl.Main |
|
spark-submit | submit a spark app |
differs with start-master.sh and start-slave.sh,here will issue new class: "spark-class org.apache.spark.deploy.SparkSubmit .." |
|
run-example | runs the examples by given a subfix class name. |
deliver to "spark-submit .." |
|
conf | |||
spark-env.sh |
misc spark cluster settings corresponding to cluster manager,e.g. SPARK_LOCAL_IP, SPARK_CLASSPATH,SPARK_MASTER_IP.. |
eg.
hadoop@GZsw04:~$ spark-daemon.sh status org.apache.spark.deploy.master.Master 1 org.apache.spark.deploy.master.Master is running.
launch.Main.java
the union entry of issuing any class/daemon,e.g.if u wanna submit a workcount app,u can do like this:
run-example JavaWordCount /file/to/count
then the concrete class named "org.apache.spark.examples.JavaWordCount" is executed by inflecting by Main.java.also,u can dig into the src of JavaWordCount for more details.
conclusions;
a.complete flow of startup master
[start-all.sh]>start-master.sh>spark-daemon.sh>spark-class.sh > launch.Main>issue app master.Master
and the slaves(workers)
[start-all.sh]>start-slaves.sh>slaves.sh>start-slave.sh>spark-daemon.sh[ worker.Worker]>same steps with above
b.it's different from hadoop,hbase.since it will supply a union entry of launch.Main to fix several params,or coming them.
c.advances step by step will reduce the reduplicate code,improves the reusability
相关推荐
da---- 2020/3/31 13:39 2-3OSPF Adjacency Relationships and Various Types of LSAs da---- 2020/3/31 13:39 2-4OSPF Stub Area and NSSA da---- 2020/3/31 13:39 2-5OSPF VCs and Inter-Area Route Filtering da-...
Laravel开发-bw-magic-relationships 第5.1条的包装
Laravel开发-laravel-request-relationships Laravel的一个模型特性,允许您通过在请求参数中定义然后自动加载模型的关系
Laravel开发-relationships 拉拉维尔的几个关系。
$ git clone https://github.com/DanielArturoAlejoAlvarez/Eloquent-ORM-Laravel-8.5-Relationships-Advanced[NAME APP] $ composer install $ copy .env.example .env $ php artisan key:generate $ ...
E88902-A Probabilistic Model of the Relationships between Countries and Climate Change.PDF
atlas-relationships-api-0.9.0-sources.jar
atlas-relationships-api-0.8.0-sources.jar
atlas-relationships-api-0.8.1-sources.jar
atlas-relationships-api-0.7.0-sources.jar
atlas-relationships-api-0.9.0.jar
atlas-relationships-api-0.8.0.jar
“#Jackson-Annotations-Domain-Relationships”
Lesson 24: A REAL-TIME APPLICATION OF MULTIPLE WAVE RELATIONSHIPS - 100 - Lesson 25: FIBONACCI TIME SEQUENCES - 105 - Lesson 26: LONG TERM WAVES - 111 - Lesson 27: THE WAVE PATTERN UP TO 1978 - 114 - ...
Graph Algorithms: Practical Examples in Apache Spark and Neo4j By 作者: Mark Needham – Amy E. Hodler ISBN-10 书号: 1492047686 ISBN-13 书号: 9781492047681 Edition 版本: 1 出版日期: 2019-01-04 pages ...
ACL 2015论文数据集 该存储库包含Nikos Voskarides,Edgar Meij,Manos Tsagkias,Maarten de Rijke和Wouter Weerkamp撰写的ACL 2015论文“学习以知识图形式说明实体关系”的数据集。 本文提出了一种自动解释实体...
破裂的关系 此自述文件概述了在此 Ember 应用程序上进行协作的详细信息。 这个应用程序的简短介绍可以很容易地转到这里。 先决条件 您将需要在您的计算机上正确安装以下东西。 (带有 NPM)和 ...
对象关系代码挑战 欢迎来到硅谷! 对于此任务,我们的领域是创业世界! 我们有三种模式- Startup , VentureCapitalist和FundingRound 。 一家Startup公司通过FundingRound拥有许多VentureCapitalist 。...
有很多实验室 目标 建立两个类之间有很多/属于的关系。 使用的构建方法在两个类之间有很多/属于。 使用类变量来跟踪有关类的数据。 构建一个类方法以公开存储在类变量中的数据。 ...在本实验中,我们将处理两组类: ...
协作对象实验室客观的深入了解对象关系。 构建通过关联和行为相互交互的类。指示这是一个测试驱动的实验室。 使用测试文件和测试输出可了解您在遵循以下准则时的要求。 在本实验中,我们将处理Artist类, Song类和...