From time to time you may find yourself in need of exporting all the topics definition from a Kafka server to recreate them on a new server. Kafka command line tools are pretty handy to do that but I could not find an already cooked script that could both dump Topics definition to a CSV file and then read it back to create topics – so I made my own two scripts.
They are pretty basic, rough and tested in standard usages, but they works. If you find any bug or have any suggestion for improvement please feel free to comment it.
First one is the script to DUMP all the Topics definition in a CSV file. The CSV file will basically contain the Topic Name, the Partitions Number, the Replication Level and any other additional Configuration Parameter related to the topic. The CSV is intentionally formatted with a Keyword before the actual value in order to do some kind of Validation in the import script. Here’s the topicsList.sh script:
#!/bin/sh
./kafka-topics.sh --describe --zookeeper localhost:2181 | grep -v "Partition: " | while read Topic Partitions Replica Configs; do
topicName=`echo $Topic | cut -d ':' -f 2`
partitionsCount=`echo $Partitions | cut -d ':' -f 2`
replicaCount=`echo $Replica | cut -d ':' -f 2`
configDetails=`echo $Configs | cut -d ':' -f 2`
echo Topic,$topicName,Partitions,$partitionsCount,Replica,$replicaCount,Configs,$configDetails
done
Here’s a sample CSV you will get with this script:
Topic,bss.order.event,Partitions,12,Replica,2,Configs,retention.ms=345600000
Topic,bss.order.event-dlt,Partitions,3,Replica,3,Configs,
Then, finally, here’s the script to import the CSV list of topics into a new Kafka server. You have to invoke it by passing the CSV file name on the command line, and in the first lines you have some toggles to limit the maximum replica you want to apply (i.e.: you’re replicating topics on a smaller cluster) and to apply some default configurations to all the topics you’re creating. Here’s topicsCreate.sh:
#!/bin/sh
## Allows you to override the max replicas we want to set-up
MAXREPLICA=1
## Allows you to specify the Default Configurations to apply
DEFAULTCONFIGS="segment.jitter.ms=5000,segment.ms=28800000,retention.bytes=524288000,segment.bytes=524288000"
if [ "$1" == "" ] || [ ! -f "$1" ]; then
echo "Please specify kafka topic file to import."
echo
echo "Format:"
echo " - one topic per line"
echo " - line format: Topic,<topic>,Partitions,<partitions>,Replica,<replica>,Configs,<configs>"
exit
fi
cat $1 | while IFS="," read -r TopicKey TopicValue PartitionKey PartitionValue ReplicaKey ReplicaValue ConfigsKey ConfigsValues; do
if [ "$TopicKey" != "Topic" ] || [ "$PartitionKey" != "Partitions" ] || [ "$ReplicaKey" != "Replica" ] || [ "$ConfigsKey" != "Configs" ]; then
echo ""
echo "Corrupt CSV file!"
echo
exit
fi
if [ $ReplicaValue -gt $MAXREPLICA ]; then
ReplicaValue=$MAXREPLICA;
fi
echo ""
echo "Creating topic $TopicValue with $PartitionValue partitions and $ReplicaValue replicas:"
./kafka-topics.sh --create --bootstrap-server localhost:9092 --topic="$TopicValue" --partitions="$PartitionValue" --replication-factor="$ReplicaValue"
echo "==> Applying default configurations for topic $TopicValue:"
./kafka-configs.sh --alter --bootstrap-server localhost:9092 --topic="$TopicValue" --add-config="$DEFAULTCONFIGS"
if [ "$ConfigsValues" != "" ]; then
echo "==> Applying custom configurations $ConfigsValues for topic $TopicValue"
./kafka-configs.sh --alter --bootstrap-server localhost:9092 --topic="$TopicValue" --add-config=$ConfigsValues
fi
done
Have fun!
extract the nectar, burn the tree 