建站
咨询
售后
建站咨询
sparksql与hive如何整合
这篇文章主要介绍sparksql与hive如何整合，文中介绍的非常详细，具有一定的参考价值，感兴趣的小伙伴们一定要看完！
创新互联专注于企业网络营销推广、网站重做改版、霍林郭勒网站定制设计、自适应品牌网站建设、H5技术、商城网站建设、集团公司官网建设、外贸网站建设、高端网站制作、响应式网页设计等建站业务，价格优惠性价比高，为霍林郭勒等各大城市提供网站开发制作服务。
hive配置
编辑 $HIVE_HOME/conf/hive-site.xml,增加如下内容:
12345 hive.metastore.uris thrift://master:9083 Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.
启动hive metastore
启动 metastore: $hive --service metastore & 查看 metastore: $jobs[1]+ Running hive --service metastore & 关闭 metastore:$kill %1kill %jobid,1代表job id1234567891011
spark配置
将 $HIVE_HOME/conf/hive-site.xml copy或者软链 到 $SPARK_HOME/conf/将 $HIVE_HOME/lib/MySQL-connector-java-5.1.12.jar copy或者软链到$SPARK_HOME/lib/copy或者软链$SPARK_HOME/lib/ 是方便spark standalone模式使用123
启动spark-sql
standalone模式
./bin/spark-sql --master spark:master:7077 --jars /home/stark_summer/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jar
1
yarn-client模式
$./bin/spark-sql --master yarn-client --jars /home/stark_summer/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jar执行 sql: select count(*) from o2o_app;结果:302Time taken: 0.828 seconds, Fetched 1 row(s)2015-09-14 18:27:43,158 INFO [main] CliDriver (SessionState.java:printInfo(536)) - Time taken: 0.828 seconds, Fetched 1 row(s) spark-sql> 2015-09-14 18:27:43,160 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - Finished stage: org.apache.spark.scheduler.StageInfo@5939ed302015-09-14 18:27:43,161 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - task runtime:(count: 1, mean: 242.000000, stdev: 0.000000, max: 242.000000, min: 242.000000)2015-09-14 18:27:43,161 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 0% 5% 10% 25% 50% 75% 90% 95% 100%2015-09-14 18:27:43,161 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 242.0 ms 242.0 ms 242.0 ms 242.0 ms 242.0 ms 242.0 ms 242.0 ms 242.0 ms 242.0 ms2015-09-14 18:27:43,162 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - fetch wait time:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)2015-09-14 18:27:43,162 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 0% 5% 10% 25% 50% 75% 90% 95% 100%2015-09-14 18:27:43,162 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms2015-09-14 18:27:43,163 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - remote bytes read:(count: 1, mean: 31.000000, stdev: 0.000000, max: 31.000000, min: 31.000000)2015-09-14 18:27:43,163 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 0% 5% 10% 25% 50% 75% 90% 95% 100%2015-09-14 18:27:43,163 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 31.0 B 31.0 B 31.0 B 31.0 B 31.0 B 31.0 B 31.0 B 31.0 B 31.0 B2015-09-14 18:27:43,163 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - task result size:(count: 1, mean: 1228.000000, stdev: 0.000000, max: 1228.000000, min: 1228.000000)2015-09-14 18:27:43,163 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 0% 5% 10% 25% 50% 75% 90% 95% 100%2015-09-14 18:27:43,163 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 1228.0 B 1228.0 B 1228.0 B 1228.0 B 1228.0 B 1228.0 B 1228.0 B 1228.0 B 1228.0 B2015-09-14 18:27:43,164 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - executor (non-fetch) time pct: (count: 1, mean: 69.834711, stdev: 0.000000, max: 69.834711, min: 69.834711)2015-09-14 18:27:43,164 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 0% 5% 10% 25% 50% 75% 90% 95% 100%2015-09-14 18:27:43,164 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 70 % 70 % 70 % 70 % 70 % 70 % 70 % 70 % 70 %2015-09-14 18:27:43,165 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - fetch wait time pct: (count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)2015-09-14 18:27:43,165 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 0% 5% 10% 25% 50% 75% 90% 95% 100%2015-09-14 18:27:43,165 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 %2015-09-14 18:27:43,166 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - other time pct: (count: 1, mean: 30.165289, stdev: 0.000000, max: 30.165289, min: 30.165289)2015-09-14 18:27:43,166 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 0% 5% 10% 25% 50% 75% 90% 95% 100%2015-09-14 18:27:43,166 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo(59)) - 30 % 30 % 30 % 30 % 30 % 30 % 30 % 30 % 30 %12345678910111213141516171819202122232425262728293031
yarn-cluster模式
./bin/spark-sql --master yarn-cluster --jars /home/dp/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jarError: Cluster deploy mode is not applicable to Spark SQL shell. Run with --help for usage help or --verbose for debug output2015-09-14 18:28:28,291 INFO [Thread-0] util.Utils (Logging.scala:logInfo(59)) - Shutdown hook called Cluster deploy mode 不支持的123456
启动 spark-shell
standalone模式
./bin/spark-shell --master spark:master:7077 --jars /home/stark_summer/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jar1
yarn-client模式
./bin/spark-shell --master yarn-client --jars /home/dp/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jarsqlContext.sql("from o2o_app SELECT count(appkey,name1,name2)").collect().foreach(println)1234以上是“sparksql与hive如何整合”这篇文章的所有内容，感谢各位的阅读！希望分享的内容对大家有帮助，更多相关知识，欢迎关注创新互联行业资讯频道！
新闻名称：sparksql与hive如何整合
文章URL：http://sczitong.cn/article/giipsj.html
