Apache Kafka를 데이터베이스에 연결하십시오.

Question

PostgreSQL, Apache Kafka (데이터 전송 용) 및 Apache Spark Streaming (데이터 처리 용)을 사용하여 스트리밍/빅 데이터 애플리케이션을 만들려고합니다.

이 document 뒤에 나는 먼저 Kafka를 Sqlite 데이터베이스에 연결하려고 시도합니다. 하지만 명령을 사용하여 커넥터를 실행할 때

나는 프로세스 종료 전에 아래와 같은 오류가 발생합니다. 내 데이터베이스에 올바른 경로를 전달합니다.

내 데이터베이스간에 연결하고 작업 할 수있는 방법 & 카프카?

2017-09-11 12:00:17,305] INFO jetty-9.2.15.v20160210 (org.eclipse.jetty.server.Server:327) 
Sep 11, 2017 12:00:17 PM org.glassfish.jersey.internal.Errors logErrors 
WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation. 
WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation. 
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation. 
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation. 

[2017-09-11 12:00:17,618] INFO Started o.e.j.s.ServletContextHandler@1a2d50d5{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744) 
[2017-09-11 12:00:17,623] WARN FAILED ServerConnector@3e37a981{HTTP/1.1}{0.0.0.0:8083}: java.net.BindException: Address already in use (org.eclipse.jetty.util.component.AbstractLifeCycle:212) 
java.net.BindException: Address already in use 
    at sun.nio.ch.Net.bind0(Native Method) 
    at sun.nio.ch.Net.bind(Net.java:433) 
    at sun.nio.ch.Net.bind(Net.java:425) 
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321) 
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) 
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236) 
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) 
    at org.eclipse.jetty.server.Server.doStart(Server.java:366) 
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) 
    at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:145) 
    at org.apache.kafka.connect.runtime.Connect.start(Connect.java:53) 
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82) 
[2017-09-11 12:00:17,624] WARN FAILED org.eclipse.jetty.server.Server@46eb180e: java.net.BindException: Address already in use (org.eclipse.jetty.util.component.AbstractLifeCycle:212) 
java.net.BindException: Address already in use 
    at sun.nio.ch.Net.bind0(Native Method) 
    at sun.nio.ch.Net.bind(Net.java:433) 
    at sun.nio.ch.Net.bind(Net.java:425) 
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321) 
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) 
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236) 
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) 
    at org.eclipse.jetty.server.Server.doStart(Server.java:366) 
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) 
    at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:145) 
    at org.apache.kafka.connect.runtime.Connect.start(Connect.java:53) 
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82) 
[2017-09-11 12:00:17,624] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:100) 
org.apache.kafka.connect.errors.ConnectException: Unable to start REST server 
    at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:147) 
    at org.apache.kafka.connect.runtime.Connect.start(Connect.java:53) 
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82) 
Caused by: java.net.BindException: Address already in use 
    at sun.nio.ch.Net.bind0(Native Method) 
    at sun.nio.ch.Net.bind(Net.java:433) 
    at sun.nio.ch.Net.bind(Net.java:425) 
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321) 
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) 
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236) 
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) 
    at org.eclipse.jetty.server.Server.doStart(Server.java:366) 
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) 
    at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:145) 
    ... 2 more 

Answer 1

WARN FAILED ServerConnector@3e37a981{HTTP/1.1}{0.0.0.0:8083}: java.net.BindException: Address already in use (org.eclipse.jetty.util.component.AbstractLifeCycle:212) 
java.net.BindException: Address already in use 

이는 Confluent Platform를 사용하고 다른 무언가가 이미 포트 8083

을 사용하고 있음을 의미? 이것은 Apache Kafka를 시작하는 좋은 방법이며 스택 (confluent start)을 가져오고 Kafka Connect를 구성하는 쉬운 방법을 제공합니다. 이 블로그 시리즈는 행동을 보여줍니다

• https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
• https://www.confluent.io/blog/blogthe-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2/
• https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/

가 BTW 당신은 ​​카프카 자체 내에서 스트림 처리를 수행하는 방법으로도 Kafka Streams을 확인 할 수 있습니다.