Mapreduce部署与第三方依赖包管理

Posted 2022-03-17 jerryzh

参考了几篇文章，总算搞定打包第三方jar了，记录一下

 

https://segmentfault.com/a/1190000016237395

https://www.cnblogs.com/Dhouse/p/6595344.html

https://yq.aliyun.com/articles/308777

https://blog.csdn.net/u013177446/article/details/54134583

 

pom.xml加入maven-assembly-plugin

                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
	                <artifactId>maven-assembly-plugin</artifactId>
	                <configuration>
<!-- 	                    <descriptorRefs> -->
<!-- 	                        <descriptorRef>jar-with-dependencies</descriptorRef> -->
<!-- 	                    </descriptorRefs> -->
	                    <descriptors>
                            <descriptor>src/assembly.xml</descriptor>
                        </descriptors>
	                </configuration>
	                <executions>
		                <execution>
		                    <id>make-assembly</id>
		                    <phase>package</phase>
		                    <goals>
		                        <goal>single</goal>
		                    </goals>
		                </execution>
		            </executions>
	            </plugin>

 

src/assembly.xml指定需要打包哪些第三方包

<?xml version="1.0" encoding="UTF-8"?>
<assembly>
    <id>cascading</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
<!--             默认为添加为.class，指定为false，则以jar包的方式导入 -->
            <unpack>false</unpack>
            <scope>runtime</scope>
            <!--
            <excludes>
                <exclude>org.apache.hadoop:*</exclude>
            </excludes>
            -->
            <!-- very small jar -->
<!--             指定路径，放到lib目录下 -->
            <outputDirectory>lib</outputDirectory>
            <includes>
                <include>fastjson:*</include>
            </includes>
        </dependencySet>
    </dependencySets>
    <fileSets>
        <fileSet>
            <directory>$project.build.outputDirectory</directory>
            <outputDirectory>/</outputDirectory>
        </fileSet>
    </fileSets>
</assembly>

 

打包

mvn assembly:assembly -Dmaven.test.skip=true

生成的jar包中，第三方jar包fastjson放在lib目录下。直接上传生成的jar到hadoop，可以直接运行