Farewell, DataSift!

Monday the 5th of September will be my last day at DataSift. Since I started here in November 2012, I’ve seen the company quadruple in size and launch several very cool technologies, such as the ability to search everything on Facebook in realtime.

Personally, I’ve learned a ton about system design, testing and building robust, usable APIs and daemons. On top of that, I’ve learned how to deploy and monitor those same services across multiple different machines and environments, as well as how to automate the entire process from a commit on my desktop to that service being released to production (yes, actual continuous deployment!). I even managed to speak at/attend more than a few conferences thanks to the things I’ve learned along the way and the conference day allowance that the company provides.

It saddens me to be leaving, but after almost four years it’s time to look for a new challenge where I can learn new skills. I’m taking a month off to drive down the west coast of the USA, but in October I’ll be returning to a job at digi.me, where I’ll be taking the title of TIMBR lead.

TIMBR stands for Tools and Infrastructure, with Managed Build and Release. I’ll be working on the automation systems and infrastructure at digi.me as well as working with the team and partners to define processes and statements that ensure that any code that we write that accesses partner systems hasn’t been compromised. This will involve work on their continuous integration platform, as well as some legal documentation that certifies the builds that we’re producing.

I’ll only be working at digi.me four days per week, taking the final day to work on personal projects such as books and conference talks. Hopefully I’ll be able to follow in the footsteps of the Grumpy Programmer himself

So, thank you DataSift for an amazing few years. You were definitely the right place for me during the formative years of my development career. I’m looking forward to taking what you taught me, iterating on it and helping share that knowledge with others too.

Building with FPM

We build RPMs for all of our internal projects at DataSift, which means that we end up writing (or more likely, copying) an RPM spec file in to each project. Some of our build tools use FPM to build themselves, and I figured it was worth a look.

Imagine that we have a project name shinything. I’ve simplified our actual project layout a lot, but at it’s core it looks like the following:

.
├── resources
├── src
│   ├── application
│   │   ├── controller
│   │   │   └── IndexController.php
│   │   ├── model
│   │   │   └── User.php
│   │   └── view
│   │       └── show-user.php
│   ├── composer.json
│   ├── composer.lock
│   ├── config
│   │   ├── mappings.json
│   │   └── targets.json
│   └── public
│       └── index.php
└── test

From this, we want everything in the src folder to be in our RPM package, making sure that the test and resources folders never make it to production. We want to install the application to /var/www/shinything and it’s config to /etc/shinything.

The first thing we need to do is to tell fpm that we want to build from a directory, we want an RPM as our output and that our RPM should be called “shinything”.

fpm -s dir -t rpm -n shinything

Next, we need to map the files on disk to their location once installed. To do this, you use the source=dest option

fpm -s dir -t rpm -n shinything src/=/var/www/shinything

This means that everything in src will be installed to /var/www/shinything. The / after src is important, as it means use everything inside that directory, without using that directory itself.

As well as the application, we need to install the config files we need to /etc/shinything. We can add another source=dest option to do this:

fpm -s dir -t rpm -n shinything src/=/var/www/shinything src/config/=/etc/shinything

We want to mark anything in /etc as a config file so that they are not overwritten when we update our RPM if there are local changes:

fpm -s dir -t rpm -n shinything --config-files /etc --exclude src/config src/=/var/www/shinything src/config/=/etc/shinything

Finally, we want to remove the config files from /var/www/shinything/config so that people don’t mistakenly use them. To do this, we can use the exclude filter. Be careful not to add a leading slash or it won’t match.

fpm -s dir -t rpm -n shinything --config-files /etc --exclude "var/www/shinything/config*" src/=/var/www/shinything src/config/=/etc/shinything

At this point, we’ve created the perfect RPM!

$ fpm -s dir -t rpm -n shinything --config-files /etc --exclude "var/www/shinything/config*" src/=/var/www/shinything src/config/=/etc/shinything 
Created package {:path=>"shinything-1.0-1.x86_64.rpm"}

If we take a look inside this RPM using the rpm tool, it will list everything that will be installed.

$ rpm -qlp shinything-1.0-1.x86_64.rpm 
/etc/shinything/mappings.json
/etc/shinything/targets.json
/var/www/shinything/application/controller/IndexController.php
/var/www/shinything/application/model/User.php
/var/www/shinything/application/view/show-user.php
/var/www/shinything/composer.json
/var/www/shinything/composer.lock
/var/www/shinything/public/index.php