在 Windows 上使用“storm jar”命令远程部署 Storm 拓扑
Posted
技术标签:
【中文标题】在 Windows 上使用“storm jar”命令远程部署 Storm 拓扑【英文标题】：Deploy Storm topology remotely using "storm jar" command on Windows 【发布时间】：2015-10-06 17:40:11 【问题描述】：我正在尝试在 Windows 上使用storm jar 命令将风暴拓扑部署到远程机器
storm jar jarName.jar mainClass
这是我的storm.yaml 文件中的内容：
storm.zookeeper.servers:
- "HostName"
nimbus.host: "HostName"
nimbus.thrift.port: 6627
但每当我尝试部署时，它总是会尝试部署到我的本地机器上。
编辑
我尝试了以下配置，但它不起作用：
conf.put(Config.NIMBUS_HOST, "HostName");
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,Arrays.asList(new String[]"HostName"));
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);
我还在命令行上尝试了 -c 选项：
storm -c nimbus.host=HostName jar jarName.jar mainClass
但它不起作用；我明白了：
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
我尝试将 -c 部分放在不同的地方，但似乎没有任何效果。
我也明白了
The field Config.NIMBUS_HOST is deprecated
当我设置时：
conf.put(Config.NIMBUS_HOST, "HostName");
我使用的是 Storm 版本 0.10.0-beta
EDIT2
在我使用后从 IDE 提交工作正常
System.setProperty("storm.jar", "path to jar");
但是，它在命令行中不起作用。
编辑3
感谢马特的帮助。我能够使用 eclipse 执行的 java 命令部署拓扑。但是，我仍然无法使用storm jar 命令进行部署。它抱怨拓扑已经存在，即使我在 Storm UI 上看不到任何东西。此外，当我使用 Eclipse 或命令行中的 Java 命令进行部署时，拓扑会成功部署。我应该如何比较执行 java 命令和storm jar 命令在做什么之间的区别？我试图在 cmd 文件的这一部分中回显变量
if %storm-command% == jar (
set STORM_OPTS=%STORM_CLIENT_OPTS% %STORM_OPTS% -Dstorm.jar=%2
echo %STORM_OPTS%
set CLASSPATH=%CLASSPATH%;%2
echo %CLASSPATH%
set CLASS=%3
set args=%4
goto start
:start
shift
if [%4] == [] goto done
set args=%args% %4
echo args: %args%
goto start
:done
echo %args%
set storm-command-arguments=%args%
)
但我没有看到任何异常。另外，我在堆栈跟踪中看到以下异常
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
通知
topologies:[]
然后它抱怨拓扑名称存在
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at mainClass.submitTopology(mainClass.java:127)
at mainClass.main(mainClass.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:613)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:549)
at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16546)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16531)
at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
... 5 more
最后，这是整个堆栈跟踪。
X:\abc\xyz\target>ERROR StatusLogger Invalid URL C:/asd/ewt/apache-storm 0.10.0-beta/log4j2/cluster.xml java.net.MalformedURLException: unknown protocol: c
at java.net.URL.<init>(URL.java:593)
at java.net.URL.<init>(URL.java:483)
at java.net.URL.<init>(URL.java:432)
at java.net.URI.toURL(URI.java:1089)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getInputFromUri(ConfigurationFactory.java:296)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:395)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:254)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:419)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:102)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
at org.apache.log4j.Category.<init>(Category.java:56)
at org.apache.log4j.Logger.<init>(Logger.java:35)
at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:59)
at org.apache.log4j.Logger.getLogger(Logger.java:39)
at mainClass.<clinit>(mainClass.java:32)
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field nimbus_uptime_secs' is unset!Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at mainClass.submitTopology(mainClass.java:127)
at mainClass.main(mainClass.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:613)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:549)
at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16546)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16531)
at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
... 5 more
顺便说一句，堆栈跟踪顶部的奇怪日志记录异常/错误是什么？这可能与storm jar 命令的奇怪行为有关吗？
如果我这样做了
java -cp jarName.jar;C:\abc\asdasd\apache-storm-0.10.0-beta\lib\* mainClass
我明白了
513 [main] INFO b.s.u.Utils - Using defaults.yaml from resources
582 [main] INFO b.s.u.Utils - Using defaults.yaml from resources
590 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload for MD
5-digest: -6100635753051151148:-8346710213172614614
591 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds []
599 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
626 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protoc
ol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at com.rbccm.th2015.storm.SampleTopologyX.submitTopology(SampleTopologyX.java:127)
at com.rbccm.th2015.storm.SampleTopologyX.main(SampleTopologyX.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName,uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:613)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:549)
at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16546)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16531)
at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
... 5 more
所以它抱怨同样的事情，但通过不同的途径。这可能与我的 YAML 有关吗？ nimbus_uptime_secs 的业务是什么？
最终编辑
好的，该休息一下了。至少目前看来，sotrm jar 命令似乎不适用于 Windows。我可以清楚地看到storm.py 尝试在Windows 中执行的命令和它尝试在Linux 中执行的命令的区别。基本上，路径在 Windows 中被搞砸了，这一定是导致所有这些奇怪异常的原因。
感谢马特提供的所有帮助和支持。我真的学到了很多。
【问题讨论】：
在我使用 System.setProperty("storm.jar", "path to jar");不能从命令行工作 它抱怨拓扑已经存在。我在风暴用户界面或风暴列表命令上看不到任何内容。在我的本地或远程机器上 所以我可以启用本地日志记录，但我应该在哪里寻找它试图上传到的集群？ 抱歉，无法使用工作聊天。拒绝访问该位置 另外，我可以看到原因：org.apache.thrift7.protocol.TProtocolException：必填字段“nimbus_uptime_secs”未设置！ 【参考方案1】：通过storm jar 运行拓扑仅适用于 Linux（但不适用于 Windows）。 “storm”命令是一个 Python 脚本，假定在 Linux 上运行（Storm 官方不支持 Window）。
但是，“storm”命令仅启动具有适当设置的常规java 命令。如果拓扑是通过 IDE 提交的，IDE 会做同样的事情。如果你使用 Eclipse，你可以按照Eclipse: export running configuration中的描述提取使用的配置
注意，需要设置JVM参数-Dstorm.jar=path\to\myJar.jar；否则，StormSubmitter 将找不到应该提交给 Storm 集群的 jar。
提取java 命令后，它应该允许您在Windows 上通过命令行提交拓扑。如果您的 IDE 不支持提取命令，您仍然可以手动将其放在一起（即使这是一个繁琐的解决方案）。
会是这样的：
java -Dstorm.jar=yourJarFile.jar
-classpath yourJarFile.jar:
/home/mjsax/.m2/repository/org/apache/storm/storm-core/0.11.0-SNAPSHOT/storm-core-0.11.0-SNAPSHOT.jar:
/home/mjsax/.m2/repository/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar:
/home/mjsax/.m2/repository/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar:
/home/mjsax/.m2/repository/org/ow2/asm/asm/4.0/asm-4.0.jar:
/home/mjsax/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:
/home/mjsax/.m2/repository/org/objenesis/objenesis/1.2/objenesis-1.2.jar:
/home/mjsax/.m2/repository/org/clojure/clojure/1.6.0/clojure-1.6.0.jar:
/home/mjsax/.m2/repository/clj-time/clj-time/0.8.0/clj-time-0.8.0.jar:
/home/mjsax/.m2/repository/joda-time/joda-time/2.3/joda-time-2.3.jar:
/home/mjsax/.m2/repository/compojure/compojure/1.1.3/compojure-1.1.3.jar:
/home/mjsax/.m2/repository/org/clojure/core.incubator/0.1.0/core.incubator-0.1.0.jar:
/home/mjsax/.m2/repository/org/clojure/tools.macro/0.1.0/tools.macro-0.1.0.jar:
/home/mjsax/.m2/repository/clout/clout/1.0.1/clout-1.0.1.jar:
/home/mjsax/.m2/repository/ring/ring-core/1.1.5/ring-core-1.1.5.jar:
/home/mjsax/.m2/repository/commons-codec/commons-codec/1.6/commons-codec-1.6.jar:
/home/mjsax/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:
/home/mjsax/.m2/repository/commons-fileupload/commons-fileupload/1.2.1/commons-fileupload-1.2.1.jar:
/home/mjsax/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:
/home/mjsax/.m2/repository/hiccup/hiccup/0.3.6/hiccup-0.3.6.jar:
/home/mjsax/.m2/repository/ring/ring-devel/1.3.0/ring-devel-1.3.0.jar:
/home/mjsax/.m2/repository/clj-stacktrace/clj-stacktrace/0.2.7/clj-stacktrace-0.2.7.jar:
/home/mjsax/.m2/repository/ns-tracker/ns-tracker/0.2.2/ns-tracker-0.2.2.jar:
/home/mjsax/.m2/repository/org/clojure/tools.namespace/0.2.4/tools.namespace-0.2.4.jar:
/home/mjsax/.m2/repository/org/clojure/java.classpath/0.2.2/java.classpath-0.2.2.jar:
/home/mjsax/.m2/repository/ring/ring-jetty-adapter/1.3.0/ring-jetty-adapter-1.3.0.jar:
/home/mjsax/.m2/repository/ring/ring-servlet/1.3.0/ring-servlet-1.3.0.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-server/7.6.13.v20130916/jetty-server-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/2.5.0.v201103041518/javax.servlet-2.5.0.v201103041518.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-continuation/7.6.13.v20130916/jetty-continuation-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-http/7.6.13.v20130916/jetty-http-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-io/7.6.13.v20130916/jetty-io-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-util/7.6.13.v20130916/jetty-util-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/ring/ring-json/0.3.1/ring-json-0.3.1.jar:
/home/mjsax/.m2/repository/cheshire/cheshire/5.3.1/cheshire-5.3.1.jar:
/home/mjsax/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.3.1/jackson-core-2.3.1.jar:
/home/mjsax/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.3.1/jackson-dataformat-smile-2.3.1.jar:
/home/mjsax/.m2/repository/tigris/tigris/0.1.1/tigris-0.1.1.jar:
/home/mjsax/.m2/repository/org/clojure/tools.logging/0.2.3/tools.logging-0.2.3.jar:
/home/mjsax/.m2/repository/com/googlecode/disruptor/disruptor/2.10.4/disruptor-2.10.4.jar:
/home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-api/2.1/log4j-api-2.1.jar:
/home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-core/2.1/log4j-core-2.1.jar:
/home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.1/log4j-slf4j-impl-2.1.jar:
/home/mjsax/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar:
/home/mjsax/.m2/repository/org/slf4j/log4j-over-slf4j/1.6.6/log4j-over-slf4j-1.6.6.jar:
/home/mjsax/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:
/home/mjsax/.m2/repository/args4j/args4j/2.0.16/args4j-2.0.16.jar
package.to.your.MainClass
【讨论】：
java -Dstorm.jar="X:\abc\xyz\ppp\ooo\zzz\target\jarName.jar" -cp X:\abc\xyz\ppp\ooo\zzz\target \classes\:X:\abc\xyz\apache-storm-0.10.0-beta\lib* z.x.c.mainClass 为什么不简单地使用“java jar”而不是“storm jar”？？ 阅读我的答案....您需要发出相同的命令 Eclipse 问题...它比您的命令更复杂。类路径必须包含更多的库... 您是否使用 Eclipse 作为 IDE？您是否尝试运行 IDE 使用的java 命令？这应该在 Windows 命令行上工作......（我现在只是好奇；））
是的，eclipse 命令在转换为 jar 文件的 java 执行时起作用。但是storm jar 命令或storm.py jar 命令不起作用。他们都抱怨拓扑名称存在以上是关于在 Windows 上使用“storm jar”命令远程部署 Storm 拓扑的主要内容，如果未能解决你的问题，请参考以下文章
夺命雷公狗---linux NO:30 linux之通过X-Shell在windows上传文件到服务器上
在 Windows 7 上将文件重命名为 md5 sum + 扩展名（使用 CMD 或 PowerShell 2013）