Kafka Connect Overview
Kafka Connect is a tool to rapidly stream events in and out of Kafka. It has a narrow focus on data ingress in and egress out of the central nervous system of modern streaming frameworks, Kafka.
Kafka Connect Sources and Sinks act as sensors on the edge of the analytics platform, loading and unloading events as they happen real time. Under the hood they are Kafka consumers and producers with a simple and elegant API that allows developers to focus on moving data to and from Kafka.
No more managing multiple components, Flume, Spark JDBC imports/exports, SQOOP, custom apps. No more Oozie XML.
Using Kafka Connect we can keep the ingest/egress layers clean and unpolluted by functional requirements.
Consolidate all your E and L from ETL or ELT in to a distributed, scalable, fault tolerant service supporting an entire organization. Data scientists can’t work their magic if they have no data.
Kafka Connect Advantages
Distributed and standalone modes. This allows it to scale down to development, testing, and small production deployments.
✔ Ability to scale up to support a large organisation’s data pipeline.
✔ Low barrier to entry and low operational overhead.
✔ Decoupled from business logic. Consumers can concentrate of business functionality, performing the `T` via Kafka Streams, Apache Flink or Spark Streaming.
Do you need custom connectors to hook it to your data sources or sinks? We can help! We have built the JDBC Sink for Confluent, the DataStax DSE certified sink for Cassandra and developed and implemented several others for clients.
Contact our sales team for more information
All our connectors, including our Certified DataStax DSE Cassandra Sink, are:
Thoroughly tested at scale with our partners.
Employ KCQL. KCQL, an elegant and simple SQL like syntax for selecting fields and routing from sources or topics to Kafka or the target system.
✔ Topic to target entity mapping
✔ Field selection
✔ Auto creation
✔ Auto evolution
✔ Error policies
Developed by experienced DevOps and Big Data specialists, we know the functionalities you want and need at enterprise level.
Need connectors for IoT?
We’ve got you covered and have connectors, sources and sinks for IoT.
✔ MQTT – Test on Mosquito
✔ CoAP – Using Californiums API with optional DTLS secure connections.
Landoop Fast Data Connect UI
All of our Connectors are also available in the Landoop Kafka Connect UI.
✔ Visualise your connect cluster sink & sources.
✔ Create new connectors with few clicks.
✔ Update & Delete connectors configuration.
✔ View workers tasks health & failures.