java.lang.ClassNotFoundException : com.datastax.spark.connector.japi.rdd.CassandraTableScanJavaRDD

Question

카산드라와 함께 카프카 스파크 스트리밍을 사용하고 있습니다. 일식을 사용하여 내 Java 클래스를 실행할 때 잘 작동하지만, 내가 maven을 사용하여 빌드하고 Spark Shell에서 실행하면 예외가 발생했습니다.

java.lang.NoClassDefFoundError: com/datastax/spark/connector/japi/rdd/CassandraTableScanJavaRDD 
    at java.lang.Class.forName0(Native Method) 
    at java.lang.Class.forName(Class.java:348) 
    at org.apache.spark.util.Utils$.classForName(Utils.scala:230) 
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:712) 
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) 
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) 
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) 
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: java.lang.ClassNotFoundException: com.datastax.spark.connector.japi.rdd.CassandraTableScanJavaRDD 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 

maven jar 플러그인을 사용해 보았지만 작동하지 않습니다. spark-shell에 인수로 모든 추가 종속성을, 또는 지정 내 pom.xml 파일 구성 당신은 모든 종속성을 포함하여 "지방 항아리"를 구축 할 필요가

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 
    <groupId>com.edureka.capstone</groupId> 
    <artifactId>spark-cassandra-demo</artifactId> 
    <version>0.0.1-SNAPSHOT</version> 

    <dependencies> 

     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 --> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.11</artifactId> 
      <version>2.2.0</version> 
      <scope>provided</scope> 
     </dependency> 

     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10 --> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-streaming_2.10</artifactId> 
      <version>2.2.0</version> 
      <scope>provided</scope> 
     </dependency> 

     <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.11 --> 
     <dependency> 
      <groupId>com.datastax.spark</groupId> 
      <artifactId>spark-cassandra-connector_2.11</artifactId> 
      <version>2.0.5</version> 
     </dependency> 

     <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector-java_2.10 --> 
     <dependency> 
      <groupId>com.datastax.spark</groupId> 
      <artifactId>spark-cassandra-connector-java_2.10</artifactId> 
      <version>1.6.0-M1</version> 
     </dependency> 

     <!-- https://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-core --> 
     <dependency> 
      <groupId>com.datastax.cassandra</groupId> 
      <artifactId>cassandra-driver-core</artifactId> 
      <version>3.3.0</version> 
     </dependency> 

     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-catalyst_2.10 --> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-catalyst_2.10</artifactId> 
      <version>2.2.0</version> 
     </dependency> 

     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 --> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-sql_2.10</artifactId> 
      <version>2.2.0</version> 
     </dependency> 

     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka --> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-streaming-kafka_2.11</artifactId> 
      <version>1.6.2</version> 
     </dependency> 

     <dependency> 
      <groupId>com.twitter</groupId> 
      <artifactId>bijection-avro_2.10</artifactId> 
      <version>0.9.2</version> 
     </dependency> 

    </dependencies> 

    <build> 
     <resources> 
      <resource> 
       <directory>${basedir}/src/main/resources</directory> 
      </resource> 
     </resources> 
     <pluginManagement> 
      <plugins> 
       <plugin> 
        <groupId>org.apache.maven.plugins</groupId> 
        <artifactId>maven-compiler-plugin</artifactId> 
        <version>3.6.2</version> 
        <configuration> 
         <source>1.8</source> 
         <target>1.8</target> 
        </configuration> 
       </plugin> 
      </plugins> 
     </pluginManagement> 
    </build> 
</project> 

Answer 1

입니다 (example 1, example 2 예를 들어, 플러그인 메이븐 어셈블리를 사용하여) .

업데이트 : 충돌을 피하려면 해당 스파크 종속성을 provided으로 표시해야합니다.