2017-04-19 7 views
0

Mesos 클러스터 배치 모드에서 spark job-sever를 구성하려고합니다. jobserver config에 spark.master = "mesos : // mesos-master : 5050"을 설정했습니다. 내가 작업 서버의 사용에 대한 컨텍스트를 생성하려고 할 때java.lang.UnsatisfiedLinkError : java.library.path에 메소가 없습니다.

, 그것은 다음과 같은 예외와 함께 실패 : 나는 변수 MESOS_NATIVE_JAVA_LIBRARY는 또한 libmesos.so 의 올바른 위치를 가리키는 환경 설정

[2017-04-19 14:09:42,346] ERROR .jobserver.JobManagerActor [] [akka://JobServer/user/jobManager-42-881e-b37be6e443dd] - Failed to create context test-context, shutting down actor 
java.lang.UnsatisfiedLinkError: no mesos in java.library.path 
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) 
    at java.lang.Runtime.loadLibrary0(Runtime.java:870) 
    at java.lang.System.loadLibrary(System.java:1122) 
    at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:54) 
    at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:79) 
    at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2485) 
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:492) 
    at spark.jobserver.context.DefaultSparkContextFactory$$anon$1.<init>(SparkContextFactory.scala:119) 
    at spark.jobserver.context.DefaultSparkContextFactory.makeContext(SparkContextFactory.scala:119) 
    at spark.jobserver.context.DefaultSparkContextFactory.makeContext(SparkContextFactory.scala:114) 
    at spark.jobserver.context.SparkContextFactory$class.makeContext(SparkContextFactory.scala:63) 
    at spark.jobserver.context.DefaultSparkContextFactory.makeContext(SparkContextFactory.scala:114) 
    at spark.jobserver.JobManagerActor$$anonfun$wrappedReceive$1.applyOrElse(JobManagerActor.scala:135) 
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) 
    at spark.jobserver.common.akka.ActorStack$$anonfun$receive$1.applyOrElse(ActorStack.scala:33) 
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) 
    at spark.jobserver.common.akka.Slf4jLogging$$anonfun$receive$1$$anonfun$applyOrElse$1.apply$mcV$sp(Slf4jLogging.scala:25) 
    at spark.jobserver.common.akka.Slf4jLogging$class.spark$jobserver$common$akka$Slf4jLogging$$withAkkaSourceLogging(Slf4jLogging.scala:34) 
    at spark.jobserver.common.akka.Slf4jLogging$$anonfun$receive$1.applyOrElse(Slf4jLogging.scala:24) 
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) 
    at spark.jobserver.common.akka.ActorMetrics$$anonfun$receive$1.applyOrElse(ActorMetrics.scala:23) 
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484) 
    at spark.jobserver.common.akka.InstrumentedActor.aroundReceive(InstrumentedActor.scala:8) 
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) 
    at akka.actor.ActorCell.invoke(ActorCell.scala:495) 
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) 
    at akka.dispatch.Mailbox.run(Mailbox.scala:224) 
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234) 
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 

, 나는 수 있어요 명령 줄에서 spark submit을 사용하여 작업을 성공적으로 제출하십시오.

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master mesos://mesos-master:5050 /pathto/spark/examples.jar 100 

즉, 내 mesos 클러스터 설정이 작동 중입니다.

spark-jobserver를 위해 특별히 수행해야하는 구성이 누락 되었습니까?

답변

0

사용자를위한 MESOS_NATIVE_JAVA_LIBRARY env 변수를 설정하고 Sudo 권한으로 작업 서버를 실행하고있었습니다.