klogd2: My second try to route Syslog messages to Kafka

It was really cool to play around with klogd but I have to confess that I’d like to have more fun. So this is my aim with klogd2.

Klogd2 is essentially a new implementation of klogd but in Java, relying on Syslog4j, as I said on klogd2’s README:

I’d want to try Syslog4j on the server side, because I know it’s a rock solid stuff and all those cool kids are using it, e.g. Graylog2.

Take a couple of minutes to get a look there, when you can. As usual, I’d really appreciate your feedback and possibly a pull request.

klogd: What about route Syslog messages to Kafka?

Today I was searching for a way to route Syslog messages to Kafka, since Syslog is the standard bucket for logs on Unix-like operational systems and there are many legacy applications which use it and cannot be changed to use something else. Unfortunately, I didn’t find anything. Therefore I decided to write something to try it.

Kafka is a pretty interesting high-throughput distributed messaging system from LinkedIn’s Data Team guys, whose aim is to serve as the foundation for LinkedIn’s activity stream and operational data processing pipeline. They have used it to handle lots of real-time data everyday and have open sourced it as an Apache project. I suggest you to take a look on its design and use cases today.

The result of my first try is klogd.

It’s a dumb simple Python program which simply listen for UDP packets on 1514 port and send them to a Kafka server. Just it. So I know, of course, there are many things to be done, because klogd is still too naive. This is just the begining.

Take a time to try it and give me your feedback. Further, fork it, hack it, and send me a pull request.

slogger: lugar de mensagens de log é no Syslog

Uma coisa importante que aprendi na convivencia com os melhores sysadmins de Linux que já trabalhei (esse e esse, por exemplo) é que lugar de log é no Syslog — e se você não está fazendo isso, hummm, há uma boa chance de you’re doing it wrong my friend.

…and in The .NET World

A melhor prática para aqueles que trabalham com .NET é escrever log com o Event Log. Os melhores sysadmins de Windows que conheço estão sempre ligados (Nagios? Zabbix?) no Event Viewer.

Então, apesar deste post se destinar a Ruby/POSIX, fica a dica do Renato Lucindo — no final desse post — para a galera de .NET/Windows também.

Bem, como a biblioteca padrão do Ruby para escrever no Syslog não é lá muito amigável, resolvi facilitar um pouco as coisas e criei a gem slogger.

Dois exemplos rápidos

Antes de mais nada, installe a gem:

$ gem install slogger

Log simples:

slogger = Slogger::Logger.new "sample_app", :debug, :local0
slogger.info "A good info"
slogger.debug "A deep info"

Log com tempo de execução:

slogger = Slogger::Logger.new "sample_app", :debug, :local0
slogger.info "A really good info preceded by spent time" do
    # do something

Depois, claro, dê uma olhada no seu Syslog, para ver as mensagens “logadas”.

A lambuja

Se você estiver usando Sinatra, por exemplo, pode usar o middleware Slogger::Rack::RequestLogger em vez do famoso Rack::CommonLogger, que usa Logger (não que o Logger seja ruim).

configure do
  slogger = Slogger::Logger.new "sample_app", :debug, :local0
  use Slogger::Rack::RequestLogger, slogger

E a dica final

Ecoando a dica do Renato Lucindo: não seja ingênuo, log everything!!!

Tá esperando o que para começar a fazer a coisa direito?