Maven - Explain in Detail

Posted 2023-03-30 Erato Rabbit

maven function:

自动 download jar and its doc, source code;

automatically download dependency;

help you compile;

help you test;

help package jar or war;

help deploy;

build your project

1. clean
2. compile. source code --> executable code, java–>class file
3. test.
4. report. Is test passed?
5. package. All class file, configuration file, into a compressed file. *.jar or *.war.
6. install. Install .jar or .war into local repo.
7. deploy. 一般不用 maven.

**
**

Concept

plugins and aim: Tool you use in maven build process is plugin.

Install Maven

是用 java 写的，command prompt 里运行 need to find JAVA_HOME.

Linux

settings Reference hadoop104 我还没配！

ENV_VAR:

Linux环境变量也称之为Shell环境变量,以下划线和字母打头,由下划线、字母(区分大小写)和数字组成,习惯上使用大写字母,例如PATH、HOSTNAME、LANG等

测试：

# note!! NOT maven -v
mvn -v

note: distinguish
**
**

Command

#生成 maven project:

mvn archetype:generate

输入maven command in command line, 必须 cd the directory where pom.xml in!

Official Document

Creating a Project

You need somewhere for your project to reside. Create a directory somewhere and start a shell in that directory. On your command line, execute the following Maven goal:

mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false

If you have just installed Maven, it may take a while on the first run. This is because Maven is downloading the most recent artifacts (plugin jars and other files) into your local repository. You may also need to execute the command a couple of times before it succeeds. This is because the remote server may time out before your downloads are complete. Don’t worry, there are ways to fix that.

You will notice that the generate goal created a directory with the same name given as the artifactId. Change into that directory.

cd my-app

What did I just do?

You executed the Maven goal archetype:generate, and passed in various parameters to that goal. The prefix archetype is the plugin that provides the goal. If you are familiar with Ant, you may conceive of this as similar to a task. This archetype:generate goal created a simple project based upon a maven-archetype-quickstart archetype. Suffice it to say for now that a plugin is a collection of goals with a general common purpose.

Introduction to the POM

https://maven.apache.org/guides/introduction/introduction-to-the-pom.html#Project_Aggregation

Poject Aggregation

Project Aggregation is similar to Project Inheritance. But instead of specifying the parent POM from the module, it specifies the modules from the parent POM. By doing so, the parent project now knows its modules. To do Project Aggregation, you must do the following:

• Change the parent POMs packaging to the value “pom”.
• Specify in the parent POM the directories of its modules (children POMs).

Example 3

The Scenario

Given the previous original artifact POMs and directory structure:

com.mycompany.app:my-app:1’s POM

<project>
	<modelVersion>4.0.0</modelVersion>
	
	<groupId>com.mycompany.app</groupId>
	<artifactId>my-app</artifactId>
    <version>1</version>
</project>

com.mycompany.app:my-module:1’s POM

<project>
    <modelVersion>4.0.0</modelVersion>
 
    <groupId>com.mycompany.app</groupId>
    <artifactId>my-module</artifactId>
    <version>1</version>
</project>

directory structure

.
 |-- my-module
 |   `-- pom.xml
 `-- pom.xml

The Solution

If we are to aggregate my-module into my-app, we would only have to modify my-app.

<packaging>pom</packaging>

<modules>
  <module>my-module</module>
</modules>

In the revised com.mycompany.app:my-app:1, the packaging section and the modules sections were added. For the packaging, its value was set to “pom”, and for the modules section, we have the element <module>my-module</module>. The value of <module> is the relative path from the com.mycompany.app:my-app:1 to com.mycompany.app:my-module:1’s POM (by practice, we use the module’s artifactId as the module directory’s name).

Now, whenever a Maven command processes the parent, that same Maven command would be ran against the module as well.

User Centre > POM Reference

POM Relaionships

One powerful aspect of Maven is its handling of project relationships: this includes dependencies (and transitive dependencies), inheritance, and aggregation (multi-module projects).

1 Dependencies

Dependency management has a long tradition of being a complicated mess for anything but the most trivial of projects. “Jarmageddon” quickly ensues as the dependency tree becomes large and complicated. “Jar Hell” follows.

依赖关系管理有着悠久的传统，除了最微不足道的项目之外，其他项目都是一个复杂的麻烦。当依赖关系树变得庞大而复杂时，*"末日 "*很快就会到来。

2 Inheritance

One powerful addition that Maven brings to build management is the concept of project inheritance.

The packaging type required to be pom for parent and aggregation (multi-module) projects.

Now we may add values to the parent POM, which will be inherited by its children. Most elements from the parent POM are inherited by its children, including:

• groupId
• version
• description
• url
• organization
• developers
• dependencyManagement
• dependencies
• properties
• build
• plugin configuration…
  • …
• …

Dependency Management

Besides inheriting certain top-level elements, parents have elements to configure values for child POMs and transitive dependencies. One of those elements is dependencyManagement.

除了继承某些顶层元素外，父类也有一些元素来配置子类POM的值和横向依赖关系。

• dependencyManagement: is used by a POM to help manage dependency information across all of its children.

Note that the version and scope of artifacts which are incorporated from transitive dependencies are also controlled by version specifications in a <dependencyManagement> section.

3 Aggregations (or Multi-Module)

A project with modules is known as a multi-module, or aggregator project. Modules are projects that this POM lists, and are executed as a group. A pom packaged project may aggregate the build of a set of projects by listing them as modules,

  <modules>
    <module>my-project</module>
    <module>another-project</module>
    <module>third-project/pom-example.xml</module>
  </modules>

You do not need to consider the inter-module dependencies yourself when listing the modules; i.e. the ordering of the modules given by the POM is not important. Maven will topologically sort the modules such that dependencies are always build before dependent modules.

你不用管module们内部的依赖关系

A final note on Inheritance v. Aggregation

Inheritance and aggregation create a nice dynamic to control builds through a single, high-level POM. You often see projects that are both parents and aggregators.

However, an aggregator project and a parent project are both POM projects, they are not one and the same and should not be confused. A POM project may be inherited from - but does not necessarily have - any modules that it aggregates. Conversely, a POM project may aggregate projects that do not inherit from it.

pom.xml

http 这些是 scheme doc

pom.xml is maven’s core configuration file, like web.xml in web project.

global variable

properties

<!--charset to use when building proj，避免中文乱码-->
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

resource plugin

<build>
<resources>
	<resource>
    </resource>    
</resources>
</build>

If dont have this plugin, src/main/resource -> target/classes. src/main/java 下的非 java 文件就直接无视了，没了。

Conventional directory stru

那么classPath 读取的就是 resource？

build process

deploy – 其实我们手动把 war 扔进 linux 里 tomcat 's webapps folder, 就完成了 deploy 了，in the most primitive way.

(win tomcat startup.bat in cmd 乱码 is because tomcat8 's encoding set is utf8 but win cmd’s is gbk)

Relationship

Dependencies

Let war depends jar

Eventually java project will become .jar, be put in web proj 's WEB-INF/lib.

Inheritance

本质：A project 's pom config inherits B project’s pom config.

aim: modify a version in super proj, effect every sub proj.

You don’t need to write version, as long as your super, or super 's super … has defined version.

Only proj whose type is pom can be a super proj.

因为整理一套版本太耗时耗力了！ 比如 A doesn’t depend on B, but need B to be some version. 这种情况太多太多了。需要长时间的摸索和调整，于是经验就沉淀在 ‘公司级父工程’

Aggregation

proj-c

顺序是，回溯安装。note: 只能是 tree structrue, 不能循环依赖 circular reference。

IDEA - Import

到 pom.xml 所在的 directory, idea 才能自动识别。不过idea里 project grade 的 Import 每个都得重设 settings - maven. moddule grade 不用。

动力节点maven入门篇20节

-DarchetypeCatalog=internal

Plugins and targets

maven core program only orders commands, 具体干活的是 plugins.

单一架构案例 All in One

单体，All source, config files, resources in One proj.

• 一个项目包含one proj
• export one war
• deploy in one Tomcat

打包部署：

SSM Integration 伪 distributed 案例

Why there 不在 super proj 写 to manage versions? 而是写在 module-environment (equals to module-common in gulimall) 里？

Because if in that super proj way, 反而要写好几遍。

super proj way suitable for that scene:

not suitable for this scene:

Microservice 案例

怎么区分 distributed verso 伪distributed? 有没有服务间通过 network 的 remote call. (还有个简单方法是看有没有 register center) (以及module 有没有自己的 XXApplication.class)

Microservice verso SOA? Microservice emphsis, every service can be deployed independently. Spring Boot conceretly implemented the 理念idea of microservice.

In this case,

yellow play the role of microservices, pink play the role of dependencies, jars who will be depended on.

Although yellow are pakaged as jar, too, they can be deployed and run!

super proj pom.xml,

<dependencyManagment>
    <dependencies>
        <dependency>
            //...
        	<artifactId>spring-cloud-dependencies</artifactId>
       	 	<version>Hoxton.SR9</version>
        	<type>pom</type>
        	<scope>import</scope>
        </dependency>
		<dependency>
            //...
            <artifactId>spring-boot-dependencies</artifactId>
            <version>2.3.6.RELEASE</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagment>

import

我想有多个爸爸，怎么办？scope import.

This artifact 里没有 java 代码的，只是声明了各种包。

由于并不 import, 所以你写错了也不知道。 sub proj 用的时候，import的时候才知道。

spring boot 打包 plugin

Speacially for spring boot buiding – plugin:

<plugin>
    <groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<!--按理说不用写version,   <spring-boot-dependencies>里有mangement-->

idea 抽风报错, 报 not found 时强制执行：

mvn clean package spring-boot:repackage -Dmaven.test.skip=true

其实强制执行后，Terminal shows up “BUILD SUCCESS” 后，idea 还是红的 ~

POM 深入

投影的就近原则

other scopes

import

见 微服务案例。

但是怎么我在 sub proj 里用 spring-cloud-starter-openfein , mvn 还说我 missing version?

因为你没写和…

版本仲裁 version arbitration

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-BmTzmaI2-1679888952018)(D:\\TempCode\\JAVA\\MD\\maven-note.assets\\image-20220813173300967.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-vNivqokn-1679888952018)(D:\\TempCode\\JAVA\\MD\\maven-note.assets\\image-20220813173319592.png)]

Profile

Spark: Job in detail

1、sc.runJob -> dagScheduler.runJob -> submitJob　　

2、DAGSchedueler::submitJob 会创建 JobSubmitted 的 event 发给内嵌类 eventProcessActor

3、eventPRocessActor 在接收到 JobSubmmitted 之后调用 processEvent 处理函数

4、job 到 stage 的转换，生成 finalStage 并提交运行，关键是调用 submitStage

5、在 SubmitStage 中会计算 stage 之间的依赖关系，依赖关系分为宽依赖和窄依赖两种

6、如果计算中发现当前的 stage 没有任何依赖或者所有的依赖都已经准备完毕，则提交 task

7、提交 task 是调用函数 submitMissingTasks 来完成

8、task 真正运行在哪个 worker 上由 TaskScheduler 来管理，也就是 submitMissingTasks 会调用 TaskScheduler::submitTasks

9、TaskSchedulerImpl 中会根据 Spark 的当前运行模式来创建相应的 backend，如果是在单机运行则创建 LocalBackend

10、LocalBackend 收到 TaskSchedulerImpl 传递进行的 ReceiveOffers 事件

11、receiverOffers -> executor.launchTask -> TaskRunner.run