EMR + DynamoDB의 워크 플로우 설정 내가 파이썬 API BOTO 사용 (DynamoDB의와 하이브와) EMR 워크 플로우를 설정하려고 Hive.createTable NoSuchMethodError JsonErrorResponseHandler

Question

가 발생합니다. Amazon EMR Console을 사용하여 수동으로 스크립트를 실행할 수 있습니다. 그러나 boto를 사용하면 테이블을 만들 때 이 실패합니다. 스크립트가 아래의 예외와 함께 실패 그러나

args1 = [u's3://us-east-1.elasticmapreduce/libs/hive/hive-script', 
     u'--base-path', 
     u's3://us-east-1.elasticmapreduce/libs/hive/', 
     u'--install-hive', 
     u'--hive-versions', 
     u'0.7.1.3'] 
args2 = [u's3://us-east-1.elasticmapreduce/libs/hive/hive-script', 
     u'--base-path', 
     u's3://us-east-1.elasticmapreduce/libs/hive/', 
     u'--hive-versions', 
     u'0.7.1.3', 
     u'--run-hive-script', 
     u'--args', 
     u'-f', 
     u's3://foo/foobar/hiveexample.sql'] 
steps = [] 
for name, args in zip(('Setup Hive','Run Hive Script'),(args1,args2)): 
    step = JarStep(name, 
        's3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar', 
        step_args=args 
        ) 
    steps.append(step) 

conn = boto.connect_emr() 
job_id = conn.run_jobflow('EpisodePlay', u's3://foo/foobar/logs/', 
          steps=steps, 
          master_instance_type='m1.small', 
          slave_instance_type='m1.small', 
          num_instances=5, 
          hadoop_version="0.20.205", 
          ami_version="2.0") 

:

다음은 EMR 워크 플로우를 설정하는 BOTO 스크립트입니다.

Hive history file=/mnt/var/lib/hive_07_1/tmp/history/hive_job_log_hadoop_201203161922_1801322338.txt 
java.lang.NoSuchMethodError: com.amazonaws.http.JsonErrorResponseHandler.<init>(Ljava/util/List;)V 
    at com.amazonaws.services.dynamodb.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:663) 
    at com.amazonaws.services.dynamodb.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:525) 
    at org.apache.hadoop.hive.dynamodb.DynamoDBClient$1.call(DynamoDBClient.java:73) 
    at org.apache.hadoop.hive.dynamodb.DynamoDBClient$1.call(DynamoDBClient.java:70) 
    at org.apache.hadoop.hive.dynamodb.DynamoDBFibonacciRetryer.runWithRetry(DynamoDBFibonacciRetryer.java:65) 
    at org.apache.hadoop.hive.dynamodb.DynamoDBClient.describeTable(DynamoDBClient.java:70) 
    at org.apache.hadoop.hive.dynamodb.DynamoDBSerDe.verifyDynamoDBWriteThroughput(DynamoDBSerDe.java:139) 
    at org.apache.hadoop.hive.dynamodb.DynamoDBSerDe.initialize(DynamoDBSerDe.java:52) 
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:199) 
    at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:253) 
    at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:484) 
    at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:455) 
    at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3159) 
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:215) 
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130) 
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) 
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063) 
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900) 
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748) 
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:171) 
    at org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:253) 
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:234) 
    at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:284) 
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:461) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156) 
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.DDLTask 
Command exiting with ret '255' 

Answer 1

나는이 문제를 해결할 수있었습니다. 올바른 AMI 버전을 사용하지 않았습니다. 콘솔에서 워크 플로를 시작했을 때 DynamoDB 연결을 지원하는 최신 AMI 버전이 선택되었지만 boto 스크립트로 시작된 경우는 아닙니다.

은이 링크를 참조하십시오 : http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/EMRforDynamoDB_PreRequisites.html

job_id = conn.run_jobflow('EpisodePlay', u's3://dfhivescript/episodePlay/logs/', 
steps=steps, 
master_instance_type='m1.small', 
slave_instance_type='m1.small', 
num_instances=5, 
hadoop_version="0.20.205", 
**ami_version="2.0.4") # Correct AMI version**