大数据Hadoopmacbookpro m1/m2 arm 编译hadoop-3.3.1

Posted 2023-03-02 笑起来贼好看

在大数据数仓实践过程中，hadoop技术栈充当了非常重要的环节，很多大数据平台，中台都是基于hive+hadoop+spark来搭建的，所以我们对于hadoop的编译，构建，安装是需要掌握的非常清楚。

而我们大数据的研发人员比较常用mac电脑来开发，mac 最新的 m1/m2 arm版本的系统在编译hive/hadoop的过程中会遇到protobuf编译失败的问题。

下面就是在编译过程中，遇到的问题，希望能给大家带来清晰的指导

在arm上编译protoc，出现异常

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile (default-cli) on project hadoop-hdfs: Missing:
[ERROR] ----------
[ERROR] 1) com.google.protobuf:protoc:exe:osx-aarch_64:3.7.1
[ERROR] 
[ERROR]   Try downloading the file manually from the project website.
[ERROR] 
[ERROR]   Then, install it using the command: 
[ERROR]       mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.7.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR] 
[ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
[ERROR]       mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.7.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR]   Path to dependency: 
[ERROR]   	1) org.apache.hadoop:hadoop-hdfs:jar:3.3.1
[ERROR]   	2) com.google.protobuf:protoc:exe:osx-aarch_64:3.7.1
[ERROR] 
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR] 
[ERROR] for artifact: 
[ERROR]   org.apache.hadoop:hadoop-hdfs:jar:3.3.1
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR]   aliyunmaven (https://maven.aliyun.com/repository/spring-plugin, releases=true, snapshots=true),
[ERROR]   alimaven (http://maven.aliyun.com/nexus/content/groups/public/, releases=true, snapshots=false)
[ERROR] 

于是去官方的issue上，找到了相关的问题，目前没有兼容m1芯片的版本，需要手动指定x86_64的版本

<plugin>
  <groupId>org.xolstice.maven.plugins</groupId>
  <artifactId>protobuf-maven-plugin</artifactId>
  <version>$protobuf-maven-plugin.version</version>
  <extensions>true</extensions>
  <configuration>
    <protocArtifact>
      com.google.protobuf:protoc:$hadoop.protobuf.version:exe:$os.detected.classifier
    </protocArtifact>
    <attachProtoSources>false</attachProtoSources>
  </configuration>
  <executions>
    <execution>
      <id>src-compile-protoc</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>compile</goal>
      </goals>
      <configuration>
        <includeDependenciesInDescriptorSet>false</includeDependenciesInDescriptorSet>
        <protoSourceRoot>$basedir/src/main/proto</protoSourceRoot>
        <outputDirectory>$project.build.directory/generated-sources/java</outputDirectory>
        <clearOutputDirectory>false</clearOutputDirectory>
        <skip>true</skip>
      </configuration>
    </execution>
    <execution>
      <id>src-test-compile-protoc</id>
      <phase>generate-test-sources</phase>
      <goals>
        <goal>test-compile</goal>
      </goals>
      <configuration>
        <protoTestSourceRoot>$basedir/src/test/proto</protoTestSourceRoot>
        <outputDirectory>$project.build.directory/generated-test-sources/java</outputDirectory>
        <clearOutputDirectory>false</clearOutputDirectory>
        <skip>true</skip>
      </configuration>
    </execution>
  </executions>
</plugin>

修改插件的 $os.detected.classifier 值 为 osx-x86_64

<plugin>
  <groupId>org.xolstice.maven.plugins</groupId>
  <artifactId>protobuf-maven-plugin</artifactId>
  <version>$protobuf-maven-plugin.version</version>
  <extensions>true</extensions>
  <configuration>
    <protocArtifact>
      com.google.protobuf:protoc:$hadoop.protobuf.version:exe:osx-x86_64
    </protocArtifact>
    <attachProtoSources>false</attachProtoSources>
  </configuration>
  <executions>
    <execution>
      <id>src-compile-protoc</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>compile</goal>
      </goals>
      <configuration>
        <includeDependenciesInDescriptorSet>false</includeDependenciesInDescriptorSet>
        <protoSourceRoot>$basedir/src/main/proto</protoSourceRoot>
        <outputDirectory>$project.build.directory/generated-sources/java</outputDirectory>
        <clearOutputDirectory>false</clearOutputDirectory>
        <skip>true</skip>
      </configuration>
    </execution>
    <execution>
      <id>src-test-compile-protoc</id>
      <phase>generate-test-sources</phase>
      <goals>
        <goal>test-compile</goal>
      </goals>
      <configuration>
        <protoTestSourceRoot>$basedir/src/test/proto</protoTestSourceRoot>
        <outputDirectory>$project.build.directory/generated-test-sources/java</outputDirectory>
        <clearOutputDirectory>false</clearOutputDirectory>
        <skip>true</skip>
      </configuration>
    </execution>
  </executions>
</plugin>

希望对正在查看文章的您有所帮助，记得关注、评论、收藏，谢谢您