Today I was searching for a way to route Syslog messages to Kafka, since Syslog is the standard bucket for logs on Unix-like operational systems and there are many legacy applications which use it and cannot be changed to use something else. Unfortunately, I didn’t find anything. Therefore I decided to write something to try it.
Kafka is a pretty interesting high-throughput distributed messaging system from LinkedIn’s Data Team guys, whose aim is to serve as the foundation for LinkedIn’s activity stream and operational data processing pipeline. They have used it to handle lots of real-time data everyday and have open sourced it as an Apache project. I suggest you to take a look on its design and use cases today.
The result of my first try is klogd.
It’s a dumb simple Python program which simply listen for UDP packets on 1514 port and send them to a Kafka server. Just it. So I know, of course, there are many things to be done, because klogd is still too naive. This is just the begining.
Take a time to try it and give me your feedback. Further, fork it, hack it, and send me a pull request.