Kafka Connect Overview

kafka-logo-no-textKafka Connect is a tool to rapidly stream events in and out of Kafka. It has a narrow focus on data ingress in and egress out of the central nervous system of modern streaming frameworks, Kafka.

Kafka Connect Sources and Sinks act as sensors on the edge of the analytics platform, loading and unloading events as they happen real time. Under the hood they are Kafka consumers and producers with a simple and elegant API that allows developers to focus on moving data to and from Kafka.

Why?

No more managing multiple components, Flume, Spark JDBC imports/exports, SQOOP, custom apps. No more Oozie XML.

Using Kafka Connect we can keep the ingest/egress layers clean and unpolluted by functional requirements.

Consolidate all your E and L from ETL or ELT in to a distributed, scalable, fault tolerant service supporting an entire organization. Data scientists can’t work their magic if they have no data.

Kafka Connect Advantages

Distributed and standalone modes. This allows it to scale down to development, testing, and small production deployments.

✔ Ability to scale up to support a large organisation’s data pipeline.
✔ Low barrier to entry and low operational overhead.
✔ Decoupled from business logic. Consumers can concentrate of business functionality, performing the `T` via Kafka Streams, Apache Flink or Spark Streaming.

Custom Connectors

Do you need custom connectors to hook it to your data sources or sinks? We can help! We have built the JDBC Sink for Confluent, the DataStax DSE certified sink for Cassandra and developed and implemented several others for clients.

Contact our sales team for more information

Our Connectors

All our connectors, including our Certified DataStax DSE Cassandra Sink, are:

Thoroughly tested at scale with our partners.

Employ KCQL. KCQL, an elegant and simple SQL like syntax for selecting fields and routing from sources or topics to Kafka or the target system.

✔ Topic to target entity mapping
✔ Field selection
✔ Auto creation
✔ Auto evolution
✔ Error policies

Developed by experienced DevOps and Big Data specialists, we know the functionalities you want and need at enterprise level.

IoT Connectors

Need connectors for IoT?

We’ve got you covered and have connectors, sources and sinks for IoT.

MQTT – Test on Mosquito
CoAP – Using Californiums API with optional DTLS secure connections.

 

Confluent Command Center Compatible

If you are using Confluents excellent Control Center for Stream Monitoring you can also use deploy our connectors via the Control Center.

 

 

 

Landoop Fast Data Connect UI

All of our Connectors are also available in the Landoop Kafka Connect UI.

✔ Visualise your connect cluster sink & sources.
✔ Create new connectors with few clicks.
✔ Update & Delete connectors configuration.
✔ View workers tasks health & failures.

Share page