sqoop 가져 오기를위한 Oozie 워크 플로가 Amazon emr hue에서 실패합니다.

Question

mysql에서 하이브 시스템으로 sqoop 가져 오기를 수행하는 워크 플로우를 만들었습니다.

sqoop 작업 생성을위한 My Sqoop 작업이 성공적으로 실행되지만 MySQL에서 하이브로 가져 오기 작업을 실행하려고 할 때 실패합니다. 여기에 로그를 첨부했습니다.

sqoop - 하이브 가져 오기 (실패한 sqoop 작업)는 두 단계로 이루어집니다.

1. 먼저 내 sq0op을 HDFS 디렉토리 (내 xml에서 언급 한 targetDir)로 가져옵니다.

2. 그런 다음이 sqoop 가져 오기의 출력이 이동되어 하이브로 가져옵니다.

oozie를 통해 내 sqoop 작업을 실행할 때 targetDir에 _SUCCESS 파일이 표시되어 sqoop 가져 오기가 성공했음을 나타냅니다. 나중 단계 (2 단계) 만 실패합니다.

나는 색상 사용자로 Oozie 워크 플로를 실행하고 있습니다.

9020 [uber-SubtaskRunner] INFO org.apache.sqoop.hive.HiveImport - Loading uploaded data into Hive 
9982 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - WARNING: Use "yarn jar" to launch YARN applications. 
10278 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - SLF4J: Class path contains multiple SLF4J bindings. 
10278 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
10278 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
10278 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 
10281 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 
12413 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - 
12413 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-2.1.0-amzn-0.jar!/hive-log4j2.properties Async: true 
13750 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:586) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at java.lang.reflect.Method.invoke(Method.java:498) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.api.SessionNotRunning 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
13751 [Thread-112] INFO org.apache.sqoop.hive.HiveImport - ... 10 more 
14098 [uber-SubtaskRunner] ERROR org.apache.sqoop.tool.CreateHiveTableTool - Encountered IOException running create table job: java.io.IOException: Hive exited with status 1 
    at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:389) 
    at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:339) 
    at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:240) 
    at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:58) 
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143) 
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) 
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) 
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) 
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236) 
    at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197) 
    at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177) 
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) 
    at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236) 
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) 
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344) 
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:380) 
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:301) 
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:187) 
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:230) 
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 

Intercepting System.exit(1) 

<<< Invocation of Main class completed <<< 

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] 

Answer 1

sqoop에서 하이브를 찾을 수 없기 때문에 오류가 발생합니다.

것은 어떤 사용자에 당신이 편집

이 Sqoop을 명령을 실행하여 데이타 노드 모두에서 하이브 ENV 변수를 설정하십시오 :

명령 줄을 입력

oozie admin -shareliblist hive 

및 tez-api-*.jar을 확인는이 .

이면 hdfs 경로 /user/oozie/share/lib/lib_DATA/hive/으로 이동하고 tez-api-*.jar을 확인하십시오. 있다면 sharelib를 업데이트하고 oozie admin -sharelibupdate을 다시 확인하고 다시 확인하십시오

Answer 2

Sqoop은 런타임 중에 Tez jar (즉, 정확하게 "org.apache.tez.dag.api.SessionNotRunning"클래스)를 찾을 수 없습니다.

필요한 경우 Tez jar를 가지고 있는지 확인하십시오.