Sunday, January 30, 2011

REST service on top of Marcuscom Tinderbox

Like many people who work on FreeBSD Ports I've been using great Marcuscom Tinderbox to verify my changes. Some time ago I started to realise that it's quite a boring task to schedule a lot of builds using web interface again and again and finally I've managed to make myself start making the process a little bit easier.

It seems that building a REST service on top on the existing is a nice way to go, so that's what I've been doing last few days and finally I have a prototype now.

As I didn't want to patch original tinderbox's webui sources, I just created a sub-directory in it called 'api' and started there. To my surprise, I wasn't able to find any ready to use solution to implement REST service in PHP (maybe I've missed something though), so ended up with a few php scripts and rewrite rules to make URLs feel RESTy. It was a quite hard part of the task since I haven't been coding in PHP for years and don't feel comfortable with it even after few days of practice.

Simultaneously I've started coding a client for which I choose Python.

Workflow

The goal was to support minimum but most important (for me at least) scenario -- testing an update for a port. Let's assume the port being tested is security/gnutls-devel. The process looks like that:

1. Produce a patch and apply it to the tree (can be done via ssh, or nfs, whatever, and could be easily scripted up)
2. Check builds we have:


(21:33) novel@fsol:~ %> tbc build
id name status current port updated
1 8.x-FreeBSD IDLE None 2011-01-30 14:05:25
(21:40) novel@fsol:~ %>


So, we see an idle build here with id 1. Not so much choices, so let's use it, but first let's see what's in the queue:


(21:40) novel@fsol:~ %> tbc queue
id username port build pri status enqueued completed
10 novel security/gnutls 8.x-FreeBSD 10 SUCCESS 2011-01-30 13:54:43 2011-01-30 14:05:27
(21:45) novel@fsol:~ %>


Now adding a port to the queue:


(21:46) novel@fsol:~ %> tbc queue add -b 1 security/gnutls-devel
(21:46) novel@fsol:~ %>


Let's check if new port build added successfully:


(21:46) novel@fsol:~ %> tbc queue
id username port build pri status enqueued completed
10 novel security/gnutls 8.x-FreeBSD 10 SUCCESS 2011-01-30 13:54:43 2011-01-30 14:05:27
11 novel security/gnutls-devel 8.x-FreeBSD 10 ENQUEUED 2011-01-30 21:46:20 None
(21:46) novel@fsol:~ %>


Cool, it's here! Now we can see that our build turned into PREPARE state:


(21:48) novel@fsol:~ %> tbc build
id name status current port updated
1 8.x-FreeBSD PREPARE None 2011-01-30 21:48:15
(21:48) novel@fsol:~ %>


All we have to do now is wait and poll for changes. Now we can see it's building:


(21:59) novel@fsol:~ %> tbc build
id name status current port updated
1 8.x-FreeBSD PORTBUILD gnutls-devel-2.11.5 2011-01-30 22:00:25
(22:00) novel@fsol:~ %>


And finally we see it's done:

(22:07) novel@fsol:~ %> tbc queue
id username port build pri status enqueued completed
11 novel security/gnutls 8.x-FreeBSD 10 SUCCESS 2011-01-30 21:46:20 2011-01-30 21:57:18
12 novel security/gnutls-devel 8.x-FreeBSD 10 SUCCESS 2011-01-30 21:50:54 2011-01-30 22:09:22
(22:15) novel@fsol:~ %> tbc queue 11


For everyone who's interested I've uploaded sources on github:



NOTICE: it's an early work in progress! The code is unstable and lacks a lot of features. Anyway, any feedback about an idea and implementation is welcome.

Saturday, January 22, 2011

lc-tools 0.1.4 released!

I'm happy to announce release of lc-tools 0.1.4.

This release is based on libcould 0.4.2 which was released not so long ago. Changes from the previous version include support of the brand new SSL handling in libcloud, some custom features for Rackspace and GoGrid drivers and minor bugfixes. Please check NEWS file for a full list of changes.

There are some non-function changes as well: lc-tools got its own website and also a tutorial.

You can install the new version using either easy_install:

easy_install lctools

or by downloading and installing it by hands:

lctools-0.1.4.tar.gz

SHA256 (lctools-0.1.4.tar.gz) = 407ce1f29f0e869219ffff2f866fb6de226b693bf925817b6b43a612120b1ff7

Happy hacking on the cloud stuff!