Flume을 사용하여 HDFS에서 Twitter 데이터를 가져 오는 중에 문제가 발생했습니다.

Question

HDFS에서 트위터 데이터를 가져 오는 중 문제가 발생했습니다. 여기

이 Env.sh 파일에서 내 flume.conf 파일

TwitterAgent.sources= Twitter 
TwitterAgent.channels= MemChannel 
TwitterAgent.sinks=HDFS 
TwitterAgent.sources.TwitterSource.type=org.apache.flume.source.twitter.TwitterSource 
TwitterAgent.sources.Twitter.channels=MemChannel 
TwitterAgent.sources.Twitter.consumerKey=xxxxxxxxxxx 
TwitterAgent.sources.Twitter.consumerSecret= xxxxxxxxxxxxxxx 
TwitterAgent.sources.Twitter.accessToken=xxxxxxxxxx 
TwitterAgent.sources.Twitter.accessTokenSecret=xxxxxxxxxxx 
TwitterAgent.sources.Twitter.keywords= hadoop,election,sports, cricket,Big data 
TwitterAgent.sinks.HDFS.channel=MemChannel 
TwitterAgent.sinks.HDFS.type=hdfs 
TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:9000/user/flume/tweets 
TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream 
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text 
TwitterAgent.sinks.HDFS.hdfs.batchSize=1000 
TwitterAgent.sinks.HDFS.hdfs.rollSize=0 
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000 
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600 
TwitterAgent.channels.MemChannel.type=memory 
TwitterAgent.channels.MemChannel.capacity=10000 
TwitterAgent.channels.MemChannel.transactionCapacity=100 

, 난 내가 데이터를 얻기 위해 아래의 명령을 사용하고 이제 경로

#FLUME_CLASSPATH="/usr/lib/flume-sources-1.0-SNAPSHOT.jar" 

이 -

[cloudera@quickstart etc]$ flume-ng agent -n TwitterAgent -c conf -f /etc/flume-ng/conf/flume.conf 

쉿 일부 로그가 필요하기는하지만 아래 오류가 발생하며 HDFS 싱크가 시작된 후에 막히고 있습니다. 구성 파일에서

16/09/25 05:18:36 WARN conf.FlumeConfiguration: Could not configure source Twitter due to: Component has no type. Cannot configure. Twitter 
org.apache.flume.conf.ConfigurationException: Component has no type. Cannot configure. Twitter 
    at org.apache.flume.conf.ComponentConfiguration.configure(ComponentConfiguration.java:76) 
    at org.apache.flume.conf.source.SourceConfiguration.configure(SourceConfiguration.java:56) 
    at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSources(FlumeConfiguration.java:567) 
    at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:346) 
    at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.access$000(FlumeConfiguration.java:213) 
    at org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:127) 
    at org.apache.flume.conf.FlumeConfiguration.<init>(FlumeConfiguration.java:109) 
    at org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfiguration(PropertiesFileConfigurationProvider.java:189) 
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:89) 
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) 
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) 
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) 
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:745) 
16/09/25 05:18:36 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [TwitterAgent] 
16/09/25 05:18:36 INFO node.AbstractConfigurationProvider: Creating channels 
16/09/25 05:18:36 INFO channel.DefaultChannelFactory: Creating instance of channel MemChannel type memory 
16/09/25 05:18:36 INFO node.AbstractConfigurationProvider: Created channel MemChannel 
16/09/25 05:18:36 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs 
16/09/25 05:18:36 INFO node.AbstractConfigurationProvider: Channel MemChannel connected to [HDFS] 
16/09/25 05:18:36 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@3963542c counterGroup:{ name:null counters:{} } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name: MemChannel}} } 
16/09/25 05:18:36 INFO node.Application: Starting Channel MemChannel 
16/09/25 05:18:36 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: MemChannel: Successfully registered new MBean. 
16/09/25 05:18:36 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: MemChannel started 
16/09/25 05:18:36 INFO node.Application: Starting Sink HDFS 
16/09/25 05:18:36 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: HDFS: Successfully registered new MBean. 
16/09/25 05:18:36 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started 

Answer 1

,

TwitterAgent.sources.Twitter.type=org.apache.flume.source.twitter.TwitterSource