While I like Cloudera Manager and the centralised dashboard, configuration and admin GUI that it provides sometimes you want to auto deploy from CI tools such as Jenkins or Teamcity. For this we can  take advantage of the Cloudera Manager API. The API has Rest, Java and Python bindings. I’ll use the Python client since I’m a bit of a DEVOPS at heart.

First install cm_api

Then simply run the following python script

This script creates a role group in the Flume service. Role groups allow for easy management of multiple instance of the same role. For example we more than likely will have multiple Flume agents up. We can create a role group, give it a configuration and assign this role group to one or many agent roles. This way we have a central group for managing configuration changes.

If the role group doesn’t exist, it’s created and it’s configuration is updated. The agent configuration file is read in from a file.

Next we add the agent role to the role group and restart.

Thats it. Now we can package this script into a Jenkins step for auto deployment.

Further more we can use the API to find out where key services are in the cluster. For example you can now combine the goodness of Flume with Kafka but you need to know where the Kafka brokers and Zookeeper servers are. As part of your deployment step you can ask Cloudera Manager where they are, build your agent configuration and deploy. I additionally use this technique for applications that need to communicate with Kafka, Flume or any other services hosted in the CDH stack.

Have fun.

Share blog