spark SQL 读取mysql中的数据日志分析

Posted 2022-04-07 小徐xfg

"F:\\Program Files\\Java\\jdk1.7.0_15\\bin\\java" -Didea.launcher.port=7539 "-Didea.launcher.bin.path=F:\\Program Files (x86)\\JetBrains\\IntelliJIDEASpark\\bin" -Dfile.encoding=UTF-8 -classpath "F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\charsets.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\deploy.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\ext\\access-bridge-64.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\ext\\dnsns.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\ext\\jaccess.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\ext\\localedata.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\ext\\sunec.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\ext\\sunjce_provider.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\ext\\sunmscapi.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\ext\\zipfs.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\javaws.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\jce.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\jfr.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\jfxrt.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\jsse.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\management-agent.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\plugin.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\resources.jar;F:\\Program Files\\Java\\jdk1.7.0_15\\jre\\lib\\rt.jar;C:\\Users\\Administrator.WIN-20160809ARI\\Desktop\\temp\\sparkSql\\Bigdata05\\target\\classes;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\scala-lang\\scala-library\\2.10.6\\scala-library-2.10.6.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\spark\\spark-core_2.10\\1.6.2\\spark-core_2.10-1.6.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\avro\\avro-mapred\\1.7.7\\avro-mapred-1.7.7-hadoop2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\avro\\avro-ipc\\1.7.7\\avro-ipc-1.7.7.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\avro\\avro-ipc\\1.7.7\\avro-ipc-1.7.7-tests.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\codehaus\\jackson\\jackson-core-asl\\1.9.13\\jackson-core-asl-1.9.13.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\codehaus\\jackson\\jackson-mapper-asl\\1.9.13\\jackson-mapper-asl-1.9.13.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\twitter\\chill_2.10\\0.5.0\\chill_2.10-0.5.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\esotericsoftware\\kryo\\kryo\\2.21\\kryo-2.21.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\esotericsoftware\\reflectasm\\reflectasm\\1.07\\reflectasm-1.07-shaded.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\esotericsoftware\\minlog\\minlog\\1.2\\minlog-1.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\objenesis\\objenesis\\1.2\\objenesis-1.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\twitter\\chill-java\\0.5.0\\chill-java-0.5.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\xbean\\xbean-asm5-shaded\\4.4\\xbean-asm5-shaded-4.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\spark\\spark-launcher_2.10\\1.6.2\\spark-launcher_2.10-1.6.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\spark\\spark-network-common_2.10\\1.6.2\\spark-network-common_2.10-1.6.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\spark\\spark-network-shuffle_2.10\\1.6.2\\spark-network-shuffle_2.10-1.6.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\fusesource\\leveldbjni\\leveldbjni-all\\1.8\\leveldbjni-all-1.8.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\fasterxml\\jackson\\core\\jackson-annotations\\2.4.4\\jackson-annotations-2.4.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\spark\\spark-unsafe_2.10\\1.6.2\\spark-unsafe_2.10-1.6.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\net\\java\\dev\\jets3t\\jets3t\\0.7.1\\jets3t-0.7.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-codec\\commons-codec\\1.3\\commons-codec-1.3.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-httpclient\\commons-httpclient\\3.1\\commons-httpclient-3.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\curator\\curator-recipes\\2.4.0\\curator-recipes-2.4.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\curator\\curator-framework\\2.4.0\\curator-framework-2.4.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\zookeeper\\zookeeper\\3.4.5\\zookeeper-3.4.5.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\jline\\jline\\0.9.94\\jline-0.9.94.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\google\\guava\\guava\\14.0.1\\guava-14.0.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\eclipse\\jetty\\orbit\\javax.servlet\\3.0.0.v201112011016\\javax.servlet-3.0.0.v201112011016.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\commons\\commons-lang3\\3.3.2\\commons-lang3-3.3.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\commons\\commons-math3\\3.4.1\\commons-math3-3.4.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\google\\code\\findbugs\\jsr305\\1.3.9\\jsr305-1.3.9.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\slf4j\\slf4j-api\\1.7.10\\slf4j-api-1.7.10.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\slf4j\\jul-to-slf4j\\1.7.10\\jul-to-slf4j-1.7.10.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\slf4j\\jcl-over-slf4j\\1.7.10\\jcl-over-slf4j-1.7.10.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\log4j\\log4j\\1.2.17\\log4j-1.2.17.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\slf4j\\slf4j-log4j12\\1.7.10\\slf4j-log4j12-1.7.10.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\ning\\compress-lzf\\1.0.3\\compress-lzf-1.0.3.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\xerial\\snappy\\snappy-java\\1.1.2.1\\snappy-java-1.1.2.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\net\\jpountz\\lz4\\lz4\\1.3.0\\lz4-1.3.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\roaringbitmap\\RoaringBitmap\\0.5.11\\RoaringBitmap-0.5.11.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-net\\commons-net\\2.2\\commons-net-2.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\typesafe\\akka\\akka-remote_2.10\\2.3.11\\akka-remote_2.10-2.3.11.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\typesafe\\akka\\akka-actor_2.10\\2.3.11\\akka-actor_2.10-2.3.11.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\typesafe\\config\\1.2.1\\config-1.2.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\io\\netty\\netty\\3.8.0.Final\\netty-3.8.0.Final.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\google\\protobuf\\protobuf-java\\2.5.0\\protobuf-java-2.5.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\uncommons\\maths\\uncommons-maths\\1.2.2a\\uncommons-maths-1.2.2a.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\typesafe\\akka\\akka-slf4j_2.10\\2.3.11\\akka-slf4j_2.10-2.3.11.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\json4s\\json4s-jackson_2.10\\3.2.10\\json4s-jackson_2.10-3.2.10.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\json4s\\json4s-core_2.10\\3.2.10\\json4s-core_2.10-3.2.10.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\json4s\\json4s-ast_2.10\\3.2.10\\json4s-ast_2.10-3.2.10.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\scala-lang\\scalap\\2.10.0\\scalap-2.10.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\scala-lang\\scala-compiler\\2.10.0\\scala-compiler-2.10.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\sun\\jersey\\jersey-server\\1.9\\jersey-server-1.9.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\asm\\asm\\3.1\\asm-3.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\sun\\jersey\\jersey-core\\1.9\\jersey-core-1.9.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\mesos\\mesos\\0.21.1\\mesos-0.21.1-shaded-protobuf.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\io\\netty\\netty-all\\4.0.29.Final\\netty-all-4.0.29.Final.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\clearspring\\analytics\\stream\\2.7.0\\stream-2.7.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\io\\dropwizard\\metrics\\metrics-core\\3.1.2\\metrics-core-3.1.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\io\\dropwizard\\metrics\\metrics-jvm\\3.1.2\\metrics-jvm-3.1.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\io\\dropwizard\\metrics\\metrics-json\\3.1.2\\metrics-json-3.1.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\io\\dropwizard\\metrics\\metrics-graphite\\3.1.2\\metrics-graphite-3.1.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\fasterxml\\jackson\\core\\jackson-databind\\2.4.4\\jackson-databind-2.4.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\fasterxml\\jackson\\core\\jackson-core\\2.4.4\\jackson-core-2.4.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\fasterxml\\jackson\\module\\jackson-module-scala_2.10\\2.4.4\\jackson-module-scala_2.10-2.4.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\scala-lang\\scala-reflect\\2.10.4\\scala-reflect-2.10.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\thoughtworks\\paranamer\\paranamer\\2.6\\paranamer-2.6.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\ivy\\ivy\\2.4.0\\ivy-2.4.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\oro\\oro\\2.0.8\\oro-2.0.8.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\tachyonproject\\tachyon-client\\0.8.2\\tachyon-client-0.8.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-lang\\commons-lang\\2.4\\commons-lang-2.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-io\\commons-io\\2.4\\commons-io-2.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\tachyonproject\\tachyon-underfs-hdfs\\0.8.2\\tachyon-underfs-hdfs-0.8.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\tachyonproject\\tachyon-underfs-s3\\0.8.2\\tachyon-underfs-s3-0.8.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\tachyonproject\\tachyon-underfs-local\\0.8.2\\tachyon-underfs-local-0.8.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\net\\razorvine\\pyrolite\\4.9\\pyrolite-4.9.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\net\\sf\\py4j\\py4j\\0.9\\py4j-0.9.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\spark-project\\spark\\unused\\1.0.0\\unused-1.0.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-client\\2.6.4\\hadoop-client-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-common\\2.6.4\\hadoop-common-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-cli\\commons-cli\\1.2\\commons-cli-1.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\xmlenc\\xmlenc\\0.52\\xmlenc-0.52.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-collections\\commons-collections\\3.2.2\\commons-collections-3.2.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-logging\\commons-logging\\1.1.3\\commons-logging-1.1.3.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-configuration\\commons-configuration\\1.6\\commons-configuration-1.6.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-digester\\commons-digester\\1.8\\commons-digester-1.8.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-beanutils\\commons-beanutils\\1.7.0\\commons-beanutils-1.7.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\commons-beanutils\\commons-beanutils-core\\1.8.0\\commons-beanutils-core-1.8.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\avro\\avro\\1.7.4\\avro-1.7.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\google\\code\\gson\\gson\\2.2.4\\gson-2.2.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-auth\\2.6.4\\hadoop-auth-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\httpcomponents\\httpclient\\4.2.5\\httpclient-4.2.5.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\httpcomponents\\httpcore\\4.2.4\\httpcore-4.2.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\directory\\server\\apacheds-kerberos-codec\\2.0.0-M15\\apacheds-kerberos-codec-2.0.0-M15.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\directory\\server\\apacheds-i18n\\2.0.0-M15\\apacheds-i18n-2.0.0-M15.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\directory\\api\\api-asn1-api\\1.0.0-M20\\api-asn1-api-1.0.0-M20.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\directory\\api\\api-util\\1.0.0-M20\\api-util-1.0.0-M20.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\curator\\curator-client\\2.6.0\\curator-client-2.6.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\htrace\\htrace-core\\3.0.4\\htrace-core-3.0.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\commons\\commons-compress\\1.4.1\\commons-compress-1.4.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\tukaani\\xz\\1.0\\xz-1.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-hdfs\\2.6.4\\hadoop-hdfs-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\mortbay\\jetty\\jetty-util\\6.1.26\\jetty-util-6.1.26.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\xerces\\xercesImpl\\2.9.1\\xercesImpl-2.9.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\xml-apis\\xml-apis\\1.3.04\\xml-apis-1.3.04.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-mapreduce-client-app\\2.6.4\\hadoop-mapreduce-client-app-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-mapreduce-client-common\\2.6.4\\hadoop-mapreduce-client-common-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-yarn-client\\2.6.4\\hadoop-yarn-client-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-yarn-server-common\\2.6.4\\hadoop-yarn-server-common-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-mapreduce-client-shuffle\\2.6.4\\hadoop-mapreduce-client-shuffle-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-yarn-api\\2.6.4\\hadoop-yarn-api-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-mapreduce-client-core\\2.6.4\\hadoop-mapreduce-client-core-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-yarn-common\\2.6.4\\hadoop-yarn-common-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\javax\\xml\\bind\\jaxb-api\\2.2.2\\jaxb-api-2.2.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\javax\\xml\\stream\\stax-api\\1.0-2\\stax-api-1.0-2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\javax\\activation\\activation\\1.1\\activation-1.1.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\javax\\servlet\\servlet-api\\2.5\\servlet-api-2.5.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\com\\sun\\jersey\\jersey-client\\1.9\\jersey-client-1.9.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\codehaus\\jackson\\jackson-jaxrs\\1.9.13\\jackson-jaxrs-1.9.13.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\codehaus\\jackson\\jackson-xc\\1.9.13\\jackson-xc-1.9.13.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-mapreduce-client-jobclient\\2.6.4\\hadoop-mapreduce-client-jobclient-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\hadoop\\hadoop-annotations\\2.6.4\\hadoop-annotations-2.6.4.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\spark\\spark-sql_2.10\\1.6.2\\spark-sql_2.10-1.6.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\spark\\spark-catalyst_2.10\\1.6.2\\spark-catalyst_2.10-1.6.2.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\codehaus\\janino\\janino\\2.7.8\\janino-2.7.8.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\codehaus\\janino\\commons-compiler\\2.7.8\\commons-compiler-2.7.8.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\parquet\\parquet-column\\1.7.0\\parquet-column-1.7.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\parquet\\parquet-common\\1.7.0\\parquet-common-1.7.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\parquet\\parquet-encoding\\1.7.0\\parquet-encoding-1.7.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\parquet\\parquet-generator\\1.7.0\\parquet-generator-1.7.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\parquet\\parquet-hadoop\\1.7.0\\parquet-hadoop-1.7.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\parquet\\parquet-format\\2.3.0-incubating\\parquet-format-2.3.0-incubating.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\org\\apache\\parquet\\parquet-jackson\\1.7.0\\parquet-jackson-1.7.0.jar;C:\\Users\\Administrator.WIN-20160809ARI\\.m2\\repository\\mysql\\mysql- connector-java\\5.1.38\\mysql-connector-java-5.1.38.jar;F:\\Program Files (x86)\\JetBrains\\IntelliJIDEASpark\\lib\\idea_rt.jar" com.intellij.rt.execution.application.AppMain JdbcRDDDemo
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/11/05 17:07:52 INFO SparkContext: Running Spark version 1.6.2
16/11/05 17:07:53 INFO SecurityManager: Changing view acls to: Administrator
16/11/05 17:07:53 INFO SecurityManager: Changing modify acls to: Administrator
16/11/05 17:07:53 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Administrator); users with modify permissions: Set(Administrator)
16/11/05 17:07:54 INFO Utils: Successfully started service 'sparkDriver' on port 59476.
16/11/05 17:07:54 INFO Slf4jLogger: Slf4jLogger started
16/11/05 17:07:55 INFO Remoting: Starting remoting
16/11/05 17:07:55 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.164.1:59489]
16/11/05 17:07:55 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 59489.
16/11/05 17:07:55 INFO SparkEnv: Registering MapOutputTracker
16/11/05 17:07:55 INFO SparkEnv: Registering BlockManagerMaster
16/11/05 17:07:55 INFO DiskBlockManager: Created local directory at C:\\Users\\Administrator.WIN-20160809ARI\\AppData\\Local\\Temp\\blockmgr-652a5955-fa4d-4228-906e-19040ceddc20
16/11/05 17:07:55 INFO MemoryStore: MemoryStore started with capacity 1807.0 MB
16/11/05 17:07:55 INFO SparkEnv: Registering OutputCommitCoordinator
16/11/05 17:07:55 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/11/05 17:07:55 INFO SparkUI: Started SparkUI at http://192.168.164.1:4040
16/11/05 17:07:56 INFO Executor: Starting executor ID driver on host localhost
16/11/05 17:07:56 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 59497.
16/11/05 17:07:56 INFO NettyBlockTransferService: Server created on 59497
16/11/05 17:07:56 INFO BlockManagerMaster: Trying to register BlockManager
16/11/05 17:07:56 INFO BlockManagerMasterEndpoint: Registering block manager localhost:59497 with 1807.0 MB RAM, BlockManagerId(driver, localhost, 59497)
16/11/05 17:07:56 INFO BlockManagerMaster: Registered BlockManager
16/11/05 17:07:56 INFO SparkContext: Starting job: collect at JdbcRDDDemo.scala:33
16/11/05 17:07:56 INFO DAGScheduler: Got job 0 (collect at JdbcRDDDemo.scala:33) with 2 output partitions
16/11/05 17:07:56 INFO DAGScheduler: Final stage: ResultStage 0 (collect at JdbcRDDDemo.scala:33)
16/11/05 17:07:56 INFO DAGScheduler: Parents of final stage: List()
16/11/05 17:07:56 INFO DAGScheduler: Missing parents: List()
16/11/05 17:07:56 INFO DAGScheduler: Submitting ResultStage 0 (JdbcRDD[0] at JdbcRDD at JdbcRDDDemo.scala:21), which has no missing parents
16/11/05 17:07:56 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1432.0 B, free 1432.0 B)
16/11/05 17:07:56 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 876.0 B, free 2.3 KB)
16/11/05 17:07:56 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:59497 (size: 876.0 B, free: 1807.0 MB)
16/11/05 17:07:56 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
16/11/05 17:07:56 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (JdbcRDD[0] at JdbcRDD at JdbcRDDDemo.scala:21)
16/11/05 17:07:56 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/11/05 17:07:57 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 1907 bytes)
16/11/05 17:07:57 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/11/05 17:07:58 INFO JdbcRDD: statement fetch size set to: -2147483648 to force MySQL streaming 
16/11/05 17:07:58 INFO JdbcRDD: closed connection
16/11/05 17:07:58 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1084 bytes result sent to driver
16/11/05 17:07:58 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 1907 bytes)
16/11/05 17:07:58 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
16/11/05 17:07:58 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1542 ms on localhost (1/2)
16/11/05 17:07:59 INFO JdbcRDD: statement fetch size set to: -2147483648 to force MySQL streaming 
16/11/05 17:07:59 INFO JdbcRDD: closed connection
16/11/05 17:07:59 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1085 bytes result sent to driver
16/11/05 17:07:59 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 1270 ms on localhost (2/2)
16/11/05 17:07:59 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
16/11/05 17:07:59 INFO DAGScheduler: ResultStage 0 (collect at JdbcRDDDemo.scala:33) finished in 2.824 s
16/11/05 17:07:59 INFO DAGScheduler: Job 0 finished: collect at JdbcRDDDemo.scala:33, took 3.105903 s
ArrayBuffer((1,efef), (2,dfefe), (3,efefe), (4,dfgfr))
16/11/05 17:07:59 INFO SparkUI: Stopped Spark web UI at http://192.168.164.1:4040
16/11/05 17:07:59 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/11/05 17:07:59 INFO MemoryStore: MemoryStore cleared
16/11/05 17:07:59 INFO BlockManager: BlockManager stopped
16/11/05 17:07:59 INFO BlockManagerMaster: BlockManagerMaster stopped
16/11/05 17:07:59 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/11/05 17:07:59 INFO SparkContext: Successfully stopped SparkContext
16/11/05 17:08:00 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/11/05 17:08:00 INFO ShutdownHookManager: Shutdown hook called
16/11/05 17:08:00 INFO ShutdownHookManager: Deleting directory C:\\Users\\Administrator.WIN-20160809ARI\\AppData\\Local\\Temp\\spark-46dfada3-5494-4d98-bcc8-7fb4b76e1d12
16/11/05 17:08:00 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.


Process finished with exit code 0