klogd2: My second try to route Syslog messages to Kafka

It was really cool to play around with klogd but I have to confess that I’d like to have more fun. So this is my aim with klogd2.

Klogd2 is essentially a new implementation of klogd but in Java, relying on Syslog4j, as I said on klogd2’s README:

I’d want to try Syslog4j on the server side, because I know it’s a rock solid stuff and all those cool kids are using it, e.g. Graylog2.

Take a couple of minutes to get a look there, when you can. As usual, I’d really appreciate your feedback and possibly a pull request.

klogd: What about route Syslog messages to Kafka?

Today I was searching for a way to route Syslog messages to Kafka, since Syslog is the standard bucket for logs on Unix-like operational systems and there are many legacy applications which use it and cannot be changed to use something else. Unfortunately, I didn’t find anything. Therefore I decided to write something to try it.

Kafka is a pretty interesting high-throughput distributed messaging system from LinkedIn’s Data Team guys, whose aim is to serve as the foundation for LinkedIn’s activity stream and operational data processing pipeline. They have used it to handle lots of real-time data everyday and have open sourced it as an Apache project. I suggest you to take a look on its design and use cases today.

The result of my first try is klogd.

It’s a dumb simple Python program which simply listen for UDP packets on 1514 port and send them to a Kafka server. Just it. So I know, of course, there are many things to be done, because klogd is still too naive. This is just the begining.

Take a time to try it and give me your feedback. Further, fork it, hack it, and send me a pull request.