When I wrote the asynchronous chef handler that I presented in the previous post, I had a little idea in mind. being able to track changes made by chef. The idea grew up a little and I now release a little dashboard I wrote. It’s still in a very early stage of development but I’ll try to present the idea behind it.
Configuration management tools are awesome. Using them, you are now managing loads of servers, reaching the pub on time and you can focus on really fun stuff. A counterpart is that they almost work in your back : changes are propagated quickly, and even if you store your cookbooks/modules in a VCS, even if you review them, you still want to know what really happens on your servers. Puppet has a really nice automatic summary part to do that job, and some people are doing cool things with it. Chef also has summary of updated resources but they don’t get back to a master like puppet do. So I wrote a little report handler to push back data to wherever you want to. It is based on the stomp protocol, to be non blocking and easily scale if you have a large number of machine. With this you just need to have a consumer that will process data to your favorite datastore.
Mcollective has been able to fire up puppetd runs for a while now, via a standalone RPC call or through the puppet commander binary (check it out, spread your load). I wanted to be able to fire up my chef clients with mcollective, to use metadata to filter what should be impacted. So I wrote a little piece of ruby, mostly based on the puppet one, to achieve this. You can now do the following :
It started like a toy, to learn a little more about mcollective agents but I finally turned into something useful (at least for me). I pushed my agent “smith” on my github account. It allows you to install or remove agents within mcollective. I usually use my configuration management tool to deploy such pieces of software but it can be useful in some case to go without it.
This post follows my previous one, dealing with the reuse of chef providers of chef in mcollective. In the comments Adam Jacob had an interesting word and when I wrote my second agent, to manage package I saw it would be a piece of cake to write a really generic agent, due to the nature of chef resource (and the way to invoke them)
It has been quite calm for a couple of months here. I have switched job, it explains why I had less time to post some things.I now work at fotolia, and I switched from puppet to chef (no troll intended, I still think puppet is a great tool, please read this).
Just a little script that enables you to launch a cluster ssh based on mcollective discovery capacities. Now you can say “I want cssh to all my openbsd boxes” in only one command.
Basing more and more stuff on mcollective means relying more and more on one of its underlying components : the activeMQ middleware, and more precisely the stomp connector. I hit a weird bug a few days ago and realized that I was not functionnaly monitoring this part of the system. The port was bound and responded to connections, subscriptions were possible but messages didn’t pass through.So I wrote this little plugin that makes this possible : it creates a random string, sends it to a queue and then reads the queue to check if the result is the same.
I recently switched from puppet daemon to mcollective commander : it kicks the “stuck in outer space puppet daemon” feature out of the way and brings me nice features (as load control). To do so I deployed the puppetd agent over my boxes.
Most people that work with puppet use a VCS : subversion, git, CVS, mercurial… Pick yours. My company uses subversion and each cmmit to the repository needs to be pulled by the master. Since I have two masters, I also want them to be synchronized. Once again it’s mcollective that comes to the rescue. I wrote a very simple agent (a 5 minutes work, to be improved) that can update a specified path. Grab it here. Once it is deployed you can use a post commit hook that calls it.
Quick post : installing a mongo server is good, monitoring it is better. I wrote a very basic check that ensure your mongo DB is healthy. Grab it on my github.
In my last post I’ve been talking about mcollective (check the new website) and mongo and how awesome it is.
I run a bunch of box under OpenBSD at $WORK and I wanted to be able to run mcollective on these too. Unfortunately, there were no package available for this OS. So I took time and with some help from landry@ I was able to build a port. It has been integrated into the github repo quickly. But to be able to use mcollective you also need to have the ruby stomp connector. As I don’t like using gems and prefer them packages, I also built a port for it. You can grab it in my github repository here.
If you read this blog you may know I daily use puppet at $WORK. Puppet is made to maintain configuration on machines, but not for one shot actions. For a couple of month I used to work with fabric for this but It had a few drawbacks, mainly because you need to maintain the list of hosts you want to act on and that it hates dead hosts, even if ghantoos dropped a link showing how to get rid of this. So I replaced it with mcollective : not exactly the same (it uses an agent instead of SSH) but it dynamicaly knows which hosts are up, allows to filter on facts (from facter, the puppet companion).
Readers of this little blog may know I’ve spent some time to have a munin setup that was tweaked to be optimized. But I’ve reached a point where my knowledge did not suffice to balance the design problems of munin. This is not a rant, I just say that when your infrastructure reaches a certain size munin reaches its limits. The pull based model, the graph generation (don’t tell me about the CGI graph, this thing never worked as expected) overloaded my management box. Talking with other peoples brought collectd to my attention so I gave it a try.