Developing Symfony applications with Docker series part I: Getting started

In this series I’m gonna share all that I’ve learned while switching from a Vagrant powered environment – running all required software in a single VirtualBox instance – to a Dockerized setup where every process runs in a separate container. But what exactly is Docker? From the Docker site:

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

Now that sounds great, doesn’t it? As a matter of fact it does, but you have to get a grip on the concept before it starts paying off. In these blog post series I’ll show you how to create a multi container Symfony application and how to get the full potential out of it. For now I’ll only focus on using Docker as develop environment. Perhaps a new series for Docker in production will follow in the near future :).

I’m a Mac OS X user so some problems I describe are related to the fact that I have to use a virtualisation layer to use Docker. If you’re happen to be on Linux, you can just skip those sections.

Installation

VirtualBox is required to run a Linux virtual machine so make sure you have a recent version installed. On the Docker site follow the installation instructions. When you’re done you should have docker, docker-compose and docker-machine binary available to you.

Create Linux virtual box

With docker-machine it’s fairly easy to create and manage a virtual machine for running Docker in. Let’s create a new instance:

If you need a box with more memory (this can happen when you have a lot of containers) you can create one with more RAM with --virtualbox-memory "2048". You’ll receive a no space left on device error when that happens.

All left to do is setting correct environment variables so Docker daemon knows how to connect our box:

Now let’s try to see if it works by listing the containers:

Create Symfony project

Now we’re ready to create a new Symfony project:

Configuring docker-compose

Docker-compose reads its configuration from a docker-compose.yml file, so create a empty one in the root of your shiny new project.

You should end up with a directory structure like this:

Install php and nginx

We’re almost there, so hang on. Obviously we’re gonna need php and a webserver so let’s install php-fpm and nginx. Never reinvent the wheel, so when you need a service containerized, always search it on Docker hub. As with bundles: there’s a container for that. We’ll use the official php and nginx images for now.

Heads up: When adding more images to your configuration take note from which image they derive. Most images extends from debian:jessie, which you probably want for your images. Docker works with a layered file system, so if all your images derive from the same parent, that will speed up the build process and also consume less space (also during transfer!).

Edit your docker-compose.yml like this:

The root element is the name of the container, you can pick whatever you like. I always try to keep these short so it’s easier (less typing) when running commands against a specific container.
The image field tells docker-compose which image we want to use for our container. The ports field allows us to expose ports on the container and forward port(s) from the container to the host, so we actually connect to the container. The value "8080:80" means we’re exposing port 80 on the container and forward it to port 8080 on the host.

You should start the containers now:

Docker will pull the images from the registry, build and start them. The -d flags tells Docker to run the containers daemonized in the background. I’ll get back on that later. When it’s done, verify if they’re both up and running:

Connecting to the box

We’ve forwarded port 8080 to our host, but connecting to localhost:8080 doesn’t work (you did try the link, didn’t you? :)). Because Docker runs in a virtual machine, we need to figure out its IP so we can connect it. Of course this isn’t very difficult:

Let’s try that IP on port 8080 and you’ll see it works: http://192.168.99.100:8080/. You’ll want to add an entry for that IP in your /etc/hosts file. Let’s pick symfony3.dev for now.

As you’ve probably discovered by now, we’re presented the default nginx page and not our shiny new Symfony application. To fix this we have to link the php container to the nginx container so they can communicate with each other. The php-fpm container needs access to our project’s php files in order to parse and serve them. Also, the nginx container requires a nginx configuration. We have to alter the Docker image and for that we need a Dockerfile.

Custom Dockerfile

The Dockerfile represents every step to be taken before the container is ready to use. Normally you would use a configuration management tool (Ansible, Puppet, Chef) to accomplish this, but in Docker you manage this via the Dockerfile.

It’s import to know that each line should contain one step. Each line creates a new layer and the number of layers is limited. One logical step per line improves the caching mechanism. For more information regarding this topic, refer to the best practices.

To configure nginx we’re going to use the nginx configuration supplied by the Symfony team. We have to copy it into the container. Create a new directory docker/nginx in the project root and add the following Dockerfile:

Create a symfony3.conf file in that docker/nginx directory as well and fill it with the following configuration:

In case you haven’t you should add symfony3.dev to your /etc/hosts file with the IP from docker-machine ip docker-tutorial.

Now let’s put it all together and update our docker-compose.yml accordingly:

Take note of the changes we’ve applied: image under nginx is replaced with build: docker/nginx which refers to the directory where the Dockerfile resides. The nginx container has a links key where we link it to the php container. Both containers have a volumes key where we mount the current directory into the container under /app path. This way the container has access to the project files.

Stop all containers and build them:

Then start them again:

When you visit http://symfony3.dev:8080/app_dev.php in your browser, you’ll see the You are not allowed to access this file. Check app_dev.php for more information. message. Remove the access check from app_dev.php and try again.

Unfortunately another well known error pops up: Failed to write cache file “/app/var/cache/dev/classes.php”..

Permissions

In my opinion the best solution to this problem is to run the console commands and php-fpm process under the same user. Without any modifications, the console commands are run under root and the php-fpm process runs under www-data. To accomplish this we also have to use a Dockerfile for the php container.

Again, stop all containers:

Create a new directory php-fpm under the docker directory. Add the following Dockerfile:

Also, add the following php-fpm.conf file:

Because I suck at naming new users I just use vagrant as my development user. Think of it as a tribute to vagrant :). The docker directory tree should be:

Now build and run the containers:

If you visit http://symfony3.dev:8080/app_dev.php now, you’ll see the Symfony welcome pages smileys at you. With this “hello world” for Symfony working we end this first post.

Next post I’ll show you how to speed up things if you’re on a Mac (the default Symfony app takes ~2000 ms to load in the current situation). Also, I’ll show you the possibilities to store your data when working with containers.

Starting your own local PHP usergroup

I’m a regular visitor of local usergroups. Especially if there’s an interesting talk and the location is within driving distance (which is not that hard to find in a small country as the Netherlands) me and my friends like to pay those meetups a visit. In the spring of 2015 when I was still working with Kristian Zondervan and Erik van Wingerden we often talked about things we wanted to do in our lives and Kristian talked a few times about starting a local PHP usergroup. A few conversations and a couple of football games further Erik also joined forces. As always with new things I was a bit sceptic, but I also gave it a shot. And so BredaPHP was born.

There were no guides on how to start your own meetup so we just did what we thought was good. Below you’ll find a summary of steps we took to get our meetup up and running.

Getting started

First things first: find a crew and audience.

Find co-organizers

When you want to start a local tech usergroup first try to find people who are willing to help with the organisation. Depending on how often you want to organise meetups, it does take a considerable amount of time. We are with three people and we have bimonthly meetups. We all work fulltime so we have to do the preparations in the evening or weekends. Also communication does take some time (twitter, email, meetup etc.).

Find audience

We registered our meetup on meetup.com. This is not free, but the prices are very reasonable. When registered, potential visitors will be notified by meetup.com about your new meetup. Once people join your meetup page, you’ll see others will follow quickly. We announced our first meetup a few months after the registration if I remember correctly. So it’s no problem to be inactive the first couple of months.

The first meetup

So you have some members on meetup.com? Let’s take it to the next level and prepare the first meetup.

Location

Every meetup needs a location, food and some cold beverages. Because nearly every company has one or more open PHP vacancies it shouldn’t be hard to find a location. We made a list of cool companies working with PHP in the area. We selected two companies and sent them a mail with:

  • who we are (BredaPHP, local usergroup for PHP developers)
  • what we do (organise meetups)
  • why we do that (meet other developers, see other companies and how they do it)
  • what they can offer us (a location for our meetup with drinks)
  • what we can offer them (a lot of potential candidates and some time for a talk about the company and how they use php)
  • requirements and planning for a meetup (start and endtime, schedule, requirements as beamer etc.)

Almost every company we asked so far were happy to host and sponsor a meetup.

Speaker

Speakers like to practice their new talks on meetups. Also, it turns out speakers are perfectly normal people, and when you ask a “famous” speaker most of them will come over and do a talk if they have the time. We just mailed some people with interesting talks we would like to see ourself. Also we used the Dutch PHP NL Slack group to find and invite speakers.

Depending where the meetup is hosted, we pick up speakers from the train station and bring them to the location. When the location is within walking distance, we don’t always do this.

Raffle

We also included a raffle in our first meetups. It’s relative easy to find prices:

  • JetBrains does sponsor licenses
  • O’Really does usergroup sponsering
  • authors often supply free copies of their work to raffle

We received some negative feedback on the raffle (visitors preferred an extra talk instead of the raffle) so we skipped in on the last meetup. Perhaps an occasional raffle will happen but we removed it from our schedule.

Next meetups

Once the first meetup is done, enjoy the positive feedback and do something with it. It’s advisable to host future meetups on recurring moments (every last Friday of the month or something like that) so people know when to expect a meetup.

Now comes the hard part: keep the meetup going. This means investing time every month. But in the end it’s all worth it. You’ll meet a lot of new people and companies. Visitors will be inspired by the meetup. After we started our meetup, two other local companies also decided to host their own development meetup, so our work didn’t go unnoticed ;).

I hope our story will help someone out there to start their own meetup!

How to automate your Mac OS X setup with Ansible

Last month I updated my development machine to the shiny new El Capitan. Unfortunately as a zsh and tmux user I ran into a very annoying bug and so I was forced to re-install my MBP with Yosemite. Because I belong to the power users which tend to customise their installations a lot and because I like automation I decided I was going to automate this cumbersome job for once and for all. Of course, with my favourite configuration management tool: Ansible.

It appears I’m not the only one and found Jeff Geerling’s excellent mac-dev-playbook repository. The README.md contains a lot of useful information and the repository is an excellent starting point. I’ve also used it as starting point for my own repository. In this post I’ll explain the most important details of my setup.

Installation of Applications and packages

I like Homebrew to install packages on my Mac so that’s also what I use in my Ansible setup. There is a ready to use Ansible role available – also created by Jeff Geerling – which I recommend. To use it:

To configure the role to suit your needs you have to edit vars/main.yml. My current version as an example:

The Dock

There is a neat utility available called dockutil which allows you to take full control over your dock. You can install it via Homebrew, so just register it under homebrew_installed_packages.

I use the following tasks and vars to fix my dock:

Vars:

Terminal

I spend a significant amount of the day in the terminal, so I’ve tweaked the default Terminal settings a lot. The cool thing is these settings can be exported via Shell > Export settings. The Ansible task to import such a file:

Mac OS X tweaks

A lot of people don’t know you can configure a lot of settings via the command line interface to the user defaults. You can read and write these settings. For some inspiration take a look at my defaults.

How I use PHP generators to make my life easier

With the release of php 5.5 we got ourself a cool new language feature: generators. If you’re new to the concept I suggest you read the excellent blog post from Anthony Ferrara about this subject. In this post I’ll show an example of how I use generators in my every day work.

The other day I was working on a import of a large batch of vacancies from a remote system via a SOAP webservice (yes, C#.NET on the other side). The service returned a container containing the current resultset and a boolean value whether there where more results. Consider the following the classical approach without generators:

There’s fundamentally not much wrong with this approach. It works. But let me show a more elegant way of solving this issue which also brings more benefits to the table as you’ll see.

The GetAllVacanciesGenerator class

Take a look at this GetAllVacanciesGenerator who’s responsibility it is to do the cumbersome fetching of the results and notice the yield keyword turning it into a Generator:

The command is much cleaner now:

As you can see the command is looking pretty again. I love this kind of improvements. It’s also easy to test this seperate generator class:

This could also be done with implementing a Iterator but I think the generator is much more readable.

Creating a custom Doctrine DBAL type the right way

Today one of my colleague’s was debugging a strange issue with Doctrine’s schema validation tool which caused our test setup to fail (we’re running app/console doctrine:schema:validate as part of our CI process). The output was the same every time:

We quickly discovered the issue was caused by the custom UUID Doctrine DBAL type we introduced lately. This type was based on some gist we found on the web:

It turned out that doing a doctrine:schema:update kept executing the same update query over and over again:

One hour of debugging later I discovered the issue originated to MySqlSchemaManager::_getPortableTableColumnDefinition where a MySQL column gets reverse-engineered into a Doctrine\DBAL\Types\Type which is used in the schema comparison tool. We’re storing the UUID in a BINARY(16) field which results in a BinaryType after reverse-engineering because Doctrine can’t tell the difference between a BinaryType and UuidType because both result in the exact same MySQL column definition. However, there is a solution to fix this.

Using DC2Type in the comment

The solution is to add a comment to the field to store the metadata in. This seems to be missing in the docs but I’ve found some JIRA issue describing the feature. We have to change our column definition so the metadata of the type doesn’t get lost:

The corrected UuidType looks as following:

Unfortunately the binary type is always reverse-engineered with $fixed = true so you have to configure the UuidType accordingly (note the options) on your entities (haven’t found a better way yet):

This comment stuff in Doctrine is not well-documented and probably a lot of people experienced the same behaviour in the schema tool so I hope I prevented some nasty debug sessions in Doctrine’s core (although you learn a lot from it).

CQRS: How to handle file uploads?

If you are like me one who tries to keep up with all the cool stuff happening in the PHP world you’ve probably noticed the buzz around Domain Driven Design and more recently Event Sourcing and CQRS. Last year Qandidate released Broadway: a project providing infrastructure and helpers for introducing CQRS and Event Sourcing into your PHP stack. It wasn’t until last month before I got the change to get my hands on it. We adopted the framework in one of the latest projects at work. And it didn’t take long before we ran into all kind of problems and questions 🙂 .

So for every question we have I’ll try to write a blogpost so others can learn. Also I’m curious about how you handle the problems I describe in these posts, so don’t hesitate to comment if you have a different opinion. Let’s dive into what should be the first in a series of post about CQRS!

The problem

In the application we’re building one requirement is that users can configure attachments to be send to a user when performing some kind of action. We’re using Symfony2 and Broadway so I our code will be very specific to these frameworks. Consider the following form:

In the controller we validate the form, construct our UploadAttachment command – which is just a DTO – by passing all the values from to form to the command bus:

And the command handler calls the appropriate method on our aggregate:

Our aggregate creates a new event:

But as you probably noticed now we run into problems because we’re passing around a UploadedFile instance in an event. Imagine how this would get stored into the event store:

Storing the complete file in the event storage is theoretically possible but we prefer to store our files not in MySQL but somewhere in a S3 bucket in the cloud. If you do your event store will grow quickly and you’ll have other challenges to wrap your head around. Keep in mind events often will be transferred by some queue like RabbitMQ.

After some digging around on the internet I found some others with the same problem. On Freenode #qandidate I also asked for advice. In general everybody stores the file in the controller or command handler and passes on the id to the event.

The solution

We’ve chosen to store the file in our controller and pass on the UUID to the command. A code example is worth a thousand words:

Drawbacks

There are a couple of drawbacks in this method:

  1. every new attachment results in a new file, this could take up a lot of storage from unused files
  2. if something goes wrong in the command handler, the file is stored already

Personally I see it as a benefit we have a history of every single attachment uploaded. We can easily go back in time and revert an erroneous upload or debug what our users did wrong in case of a problem.

By only passing around the UUID our event keeps small and this makes it easy to be published on RabbitMQ.

Debugging Selenium with X11 Forwarding on Scrutinizer CI

Last week we ran into some issues on Scrutinizer CI with our Behat Selenium test suite. These things tend to be quite hard to debug: as Selenium is running headless in X virtual framebuffer (Xvfb) there is nothing to see for the developer. It’s possible to take screenshots, but this requires code changes (probably).

X11 Forwarding

One of the cool things you can do with Selenium (or more particular Firefox or Chrome) is X11 forwarding. If you’re running a X.Org Window system it is possible to forward the display from one box to another. When you’re on a Linux desktop environment you’re golden, but if you’re on a Mac like me you have to install XQuartz to get it working. Follow the instructions on the site and don’t forget to logout and login after installation otherwise your $DISPLAY enviroment variable is empty. By the way: if you’re a Windows user I honestly suggest to get a Mac or install Ubuntu to get it working :).

If you want more information on this topic there is plenty of information to be found on the interwebz.

SSH Remote debugging session

To get this X11 forwarding working on Scrutinizer CI first request a new SSH debugging session. This can be done on the inspection page. When the inspection fails you can retry but in the same dropdown also choose for “SSH Remote debugging”. The first time you do this it will ask for your public keys from Github, you should accept the request. It may take some time but after a few seconds or minutes you’ll receive a SSH login to connect to. Add an -X switch after ssh, so the command looks like this:

Now login on the remote machine and verify your $DISPLAY server is set (should be something like localhost:10.0). Open ~/.profile in your favourite editor and remove the line:

Do not log out, but create new SSH session with the command above. If all is fine you should be able to start firefox and see the browser appearing. Kill this firefox instance, and start Selenium in this session (java -jar /location/of/selenium.jar) and in the first session start your Behat tests.

Speedup your test suite on Codeship using ParallelCI

As I mentioned in an earlier blog post we use Codeship to test some of our private repositories. The folks at Codeship improved their service a lot since we first used it: the UI is improved a lot (both visually as practically) and the list of notification services keeps growing too.

Lately they introduced a cool new feature called ParallelCI. Travis CI has a similar feature called build matrix. You can split up your test suite in multiple parallel builds, called pipelines. If you have a big or slow test suite (probably your Behat tests) you can speed up things a lot by splitting them into multiple pipelines.

Example configuration

Because our phpspec suite is fast enough, we’ve splitted our Behat suite in multiple pipelines. Of course this is project dependant and will vary per use case. To enable ParallelCI open your project settings at Codeship and click on the “Test” link. Scroll down for the “Configure Test Pipelines” section. There will be one pipeline configured called “Test commands” in which all your current test commands are configured.
Click on the green “Add new pipeline” link and a new pipeline tab will be added. Give it a clear name and add your test command. To get an idea of how this can be done take a look at our configuration:

Tab #1: Behat user

Tab #2: Behat profile

Tab #3: phpspec

When you save these settings (the pipeline edit form is bit cumbersome as you will notice when adding new tabs, but I guess this will be improved soon enough) and rerun your last run you’ll see your suite will be split into multiple pipelines and as a result it will speedup things drastically. So I definitely see the use of this new feature and I’m sure you’ll love it for your bigger test suites.

Symfony2 and RabbitMQ: Lessons learned

Last year we introduced RabbitMQ into our stack at Waarneembemiddeling.nl. We were in desperate need of a worker queue and after fiddling around with Gearman, Beanstalkd and RabbitMQ we made our choice: RabbitMQ it will be.

Now there’s quite some information to be found on RabbitMQ and how to use it, but a lot of things you have to find out yourself. Questions like:

  • what happens to messages on the queue after a service restart?
  • what happens to messages on the queue after a reboot?
  • how do we notice that a worker crashed?
  • what happens to my message when the consumer dies while processing?
  • etc.

Using RabbitMQ and Symfony2 (or php in general) is quite easy. There is a bundle for Symfony2 called OldSoundRabbitMqBundle and a php library called php-amqplib which work very well. Both are from the same author, you should probably thank him for that 🙂 .

First try: pure php consumers

We’re running a fairly common setup. Because we’ve been warned that php consumer die out every now and then, we’re using Supervisor to start new consumers when needed. There is a lot of information out there on this subject so I won’t go in there.

Despite the warnings we started with pure php consumers powered by the commands in OldSoundRabbitMqBundle. The first workers were started like this:

This means we’re consuming from the async_event queue without any limit to the messages. Basically this means it will run forever, or better said: until php crashes. Or worse: your consumer ends up in non-response state. Which means it doesn’t process any message any more and Supervisor thinks all is fine because you still have a running process. This happened once to our mail queue. I can assure you it’s better to prevent these kind of things.

Second try: pure php consumers with limited messages

So after the mail-gate I was searching for a quick way to make our setup more error proof. The OldSoundRabbitMqBundle supports limiting the messages to process. So I limited our workers so that they got restarted a couple of times a day:

After that things got running more smoothly and it took a while before we encountered new problems. While spitting trough the logs I notices some consumers produced some errors. A brief summary:

  • General error: 2006 MySQL server has gone away
  • Warning: Error while sending QUERY packet.

Because the consumer is one process that keeps running, that also means that the service container and stuff keeps existing in memory. When you’ve done some queries the database connection keeps open in the background. And if it’s quiet on our queue, it may take some time before we reach the message limit. If that time exceeds the connect_timeout of your MySQL server, you’ll run into the warnings and errors about lost connections.

Of course we should close the connection after the message is processed or could try catch for Doctrine DBAL connection exceptions or increase the connect_timeout setting but thats just denying the real problem. Running consumers with a booted Symfony2 kernel just doesn’t work so well.

A final resort could be to strip down the consumers and don’t use the Symfony2 kernel and container but we don’t like that. Our messages are most of the time serialized events which get dispatched again after the consumer picks them up. At the application level we don’t want to know wether we are in a RabbitMQ consumer or in a normal HTTP request.

Real solution: rabbitmq-cli-consumer

So it took a couple of months to learn the hard way we needed some different solution for our consumers. I found this interesting blog post about the same problem. He solved it with Java and Ruby consumers. We all learned java in college right, but I don’t like to run the memory eating jvm on our servers. The Ruby consumer unfortunately misses some good documenten for me as Ruby virgin. So I got a bit lost there.

That was the point where Go got in. Go is a kind of improved C with not real OO but a lot of cool stuff in it. I wrote a application that makes it possible to consume messages from RabbitMQ queue and pipe them into an command line application. I called it: rabbitmq-cli-consumer.

The main advantages for using rabbitmq-cli-consumer are:

  • no more stability issues to deal with
  • lightweight and fast
  • no need to restart your workers after a fresh deployment

We still use supervisor to start and stop the consumers because it’s the right tool for it. An example of how we start a consumer:

An example of a Symfony2 command we use:

Final tip: use the management plugin

Before even starting with RabbitMQ make sure you have the management plugin installed. It gives you a good overview about whats happening. Also you can purge queues, add users, add vhosts etc.

Install Selenium headless on Debian Wheezy (optionally with Ansible)

When you start testing with Behat and Mink Selenium2 driver you also need a browser running. Because we develop on a virtualised server installing FireFox was a bit more tricky then I expected. Of course a search yielded some interesting results, but also a lot of crap. So here is a little writeup on how I managed to get it running to save you some time. An example playbook can be found at the bottom of this post. But beware: this is Debian only!

On Debian there is a package called iceweasel. This is a rebranded version of FireFox. Because of that there is no firefox package available in the default repositories.

We are using Ansible for configuration management (for both our production and develop environment) so I prefer a package above compiling shit because that’s much easier to automate. There are a couple of options to install FireFox trough package manager:

  1. add Linux Mint repository
  2. add ubuntuzilla repository

Using the Linux Mint repository I experienced some problems. The Ubuntuzilla repository worked like a charm. If you want to install manually just follow the instructions in their Wiki. After adding the repository you can install the firefox package:

To run Firefox headless you also need some display server and to emulate that we are going to use xvfb. Selenium requires Java, thus we install:

Download Selenium somewhere:

You should be able to start Selenium now:

Starting by hand is a bit lame, so we use this startup script:

Copy this to /etc/init.d/selenium and after that you can:

And when we create an Ansible playbook for all this we get: