스파크 제출 "서비스 '드라이버'포트에 바인딩하지 못했습니다"오류

Question

나는 단어 수의 스파크 자바의 예를 실행하려면 다음 명령을 사용 : - 나는 그것을 실행하면 다음과 같은 출력이

time spark-submit --deploy-mode cluster --master spark://192.168.0.7:6066 --class org.apache.spark.examples.JavaWordCount /home/pi/Desktop/example/new/target/javaword.jar /books_50.txt 

입니다 : -

Running Spark using the REST application submission protocol. 
16/07/18 03:55:41 INFO rest.RestSubmissionClient: Submitting a request to launch an application in spark://192.168.0.7:6066. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Submission successfully created as driver-20160718035543-0000. Polling submission state... 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Submitting a request for the status of submission driver-20160718035543-0000 in spark://192.168.0.7:6066. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: State of driver driver-20160718035543-0000 is now RUNNING. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Driver is running on worker worker-20160718041005-192.168.0.12-42405 at 192.168.0.12:42405. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Server responded with CreateSubmissionResponse: 
{ 
    "action" : "CreateSubmissionResponse", 
    "message" : "Driver successfully submitted as driver-20160718035543-0000", 
    "serverSparkVersion" : "1.6.2", 
    "submissionId" : "driver-20160718035543-0000", 
    "success" : true 
} 

나는 로그의 특정 근로자 (192.168.0.12)를 확인하고는 말합니다 : -

Launch Command: "/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/lib/spark-assembly-1.6.2-hadoop2.6.0.jar:/opt/spark/lib/datanucleus-api-jdo-3.2.6.jar:/opt/spark/lib/datanucleus-core-3.2.10.jar:/opt/spark/lib/datanucleus-rdbms-3.2.9.jar" "-Xms1024M" "-Xmx1024M" "-Dspark.driver.supervise=false" "-Dspark.app.name=org.apache.spark.examples.JavaWordCount" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/home/pi/Desktop/example/new/target/javaword.jar" "-Dspark.master=spark://192.168.0.7:7077" "-Dspark.executor.memory=10M" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.0.12:42405" "/opt/spark/work/driver-20160718035543-0000/javaword.jar" "org.apache.spark.examples.JavaWordCount" "/books_50.txt" 
======================================== 

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
16/07/18 04:10:58 INFO SecurityManager: Changing view acls to: pi 
16/07/18 04:10:58 INFO SecurityManager: Changing modify acls to: pi 
16/07/18 04:10:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(pi); users with modify permissions: Set(pi) 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'Driver' failed after 16 retries! Consider explicitly setting the appropriate port for the service 'Driver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries. 
    at sun.nio.ch.Net.bind0(Native Method) 
    at sun.nio.ch.Net.bind(Net.java:433) 
    at sun.nio.ch.Net.bind(Net.java:425) 
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) 
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485) 
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089) 
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430) 
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415) 
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903) 
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198) 
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348) 
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) 
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 
    at java.lang.Thread.run(Thread.java:745) 

내 spark-env.sh (마스터 용) 포함 파일 : - (근로자)

export SPARK_MASTER_WEBUI_PORT="8080" 
export SPARK_MASTER_IP="192.168.0.7" 
export SPARK_EXECUTOR_MEMORY="10M" 

내 spark-env.sh 파일이 포함되어 있습니다 -

export SPARK_WORKER_WEBUI_PORT="8080" 
export SPARK_MASTER_IP="192.168.0.7" 
export SPARK_EXECUTOR_MEMORY="10M" 

도와주세요 ...!

Answer 1

이것은 내가 같은 단계를 시도했지만 작업을 실행할 수있는 가능성이 Spark 1.2.1 standalone cluster mode spark-submit is not working

의 중복입니다. 가능한 경우 전체 spark-env.sh 및 스파크 기본값을 게시하십시오.

Answer 2

/etc/hosts 파일에 호스트 이름을 입력해야합니다. 같은 뭔가 :

127.0.0.1 localhost "hostname" 

Answer 3

나는 쉘을 실행하려고 할 때 같은 문제가 있었다, 그리고 SPARK_LOCAL_IP 환경 변수를 설정하여이 작업을 얻을 수 있었다. 좀 더 영구적 인 해결책을 위해

SPARK_LOCAL_IP=127.0.0.1 ./bin/spark-shell

당신의 불꽃 루트의 conf 디렉토리에 spark-env.sh 파일을 만들 : 당신은 쉘을 실행할 때 명령 줄에서이 작업을 할당 할 수 있습니다. 다음 줄을 추가합니다 :

SPARK_LOCAL_IP=127.0.0.1

이 chmod +x ./conf/spark-env.sh를 사용하여 스크립트에 실행 권한을주고, 이것은 기본적으로이 환경 변수를 설정합니다.

Answer 4

종속성을 관리하기 위해 Maven/SBT를 사용 중이며 Spark 코어가 jar 파일에 포함되어 있습니다.

당신은 (여기 스칼라)은 "spark.driver.bindAddress"을 설정하여 런타임에 SPARK_LOCAL_IP을 재정의 할 수

val config = new SparkConf() 
config.setMaster("local[*]") 
config.setAppName("Test App") 
config.set("spark.driver.bindAddress", "127.0.0.1") 
val sc = new SparkContext(config)