Developing Symfony applications with Docker series part I: Getting started

In this series I’m gonna share all that I’ve learned while switching from a Vagrant powered environment – running all required software in a single VirtualBox instance – to a Dockerized setup where every process runs in a separate container. But what exactly is Docker? From the Docker site:

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

Now that sounds great, doesn’t it? As a matter of fact it does, but you have to get a grip on the concept before it starts paying off. In these blog post series I’ll show you how to create a multi container Symfony application and how to get the full potential out of it. For now I’ll only focus on using Docker as develop environment. Perhaps a new series for Docker in production will follow in the near future :).

I’m a Mac OS X user so some problems I describe are related to the fact that I have to use a virtualisation layer to use Docker. If you’re happen to be on Linux, you can just skip those sections.

Installation

VirtualBox is required to run a Linux virtual machine so make sure you have a recent version installed. On the Docker site follow the installation instructions. When you’re done you should have docker, docker-compose and docker-machine binary available to you.

Create Linux virtual box

With docker-machine it’s fairly easy to create and manage a virtual machine for running Docker in. Let’s create a new instance:

$ docker-machine create docker-tutorial --driver virtualbox
Running pre-create checks...
Creating machine...
(docker-tutorial) Creating VirtualBox VM...
(docker-tutorial) Creating SSH key...
(docker-tutorial) Starting VM...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect Docker to this machine, run: docker-machine env docker-tutorial

$ docker-machine create docker-tutorial --driver virtualbox

Running pre-create checks...

Creating machine...

(docker-tutorial) Creating VirtualBox VM...

(docker-tutorial) Creating SSH key...

(docker-tutorial) Starting VM...

Waiting for machine to be running, this may take a few minutes...

Machine is running, waiting for SSH to be available...

Detecting operating system of created instance...

Detecting the provisioner...

Provisioning with boot2docker...

Copying certs to the local machine directory...

Copying certs to the remote machine...

Setting Docker configuration on the remote daemon...

Checking connection to Docker...

Docker is up and running!

To see how to connect Docker to this machine, run: docker-machine env docker-tutorial

If you need a box with more memory (this can happen when you have a lot of containers) you can create one with more RAM with --virtualbox-memory "2048". You’ll receive a no space left on device error when that happens.

All left to do is setting correct environment variables so Docker daemon knows how to connect our box:

$ eval "$(docker-machine env docker-tutorial)"

1	$ eval "$(docker-machine env docker-tutorial)"

Now let’s try to see if it works by listing the containers:

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

1 2	$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

Create Symfony project

Now we’re ready to create a new Symfony project:

$ symfony new docker-tutorial

 Downloading Symfony...

    4.94 MB/4.94 MB ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  100%

 Preparing project...

 ✔  Symfony 3.0.1 was successfully installed. Now you can:

    * Change your current directory to /Users/richard/projects/docker-tutorial

    * Configure your application in app/config/parameters.yml file.

    * Run your application:
        1. Execute the php bin/console server:run command.
        2. Browse to the http://localhost:8000 URL.

    * Read the documentation at http://symfony.com/doc

$ symfony new docker-tutorial

Downloading Symfony...

4.94 MB/4.94 MB ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 100%

Preparing project...

✔ Symfony 3.0.1 was successfully installed. Now you can:

* Change your current directory to /Users/richard/projects/docker-tutorial

* Configure your application in app/config/parameters.yml file.

* Run your application:

1. Execute the php bin/console server:run command.

2. Browse to the http://localhost:8000 URL.

* Read the documentation at http://symfony.com/doc

Configuring docker-compose

Docker-compose reads its configuration from a docker-compose.yml file, so create a empty one in the root of your shiny new project.

You should end up with a directory structure like this:

.
├── app
├── bin
├── src
├── tests
├── var
├── vendor
├── web
├── README.md
├── composer.json
├── composer.lock
├── docker-compose.yml
└── phpunit.xml.dist

├── app

├── bin

├── src

├── tests

├── var

├── vendor

├── web

├── README.md

├── composer.json

├── composer.lock

├── docker-compose.yml

└── phpunit.xml.dist

Install php and nginx

We’re almost there, so hang on. Obviously we’re gonna need php and a webserver so let’s install php-fpm and nginx. Never reinvent the wheel, so when you need a service containerized, always search it on Docker hub. As with bundles: there’s a container for that. We’ll use the official php and nginx images for now.

Heads up: When adding more images to your configuration take note from which image they derive. Most images extends from debian:jessie, which you probably want for your images. Docker works with a layered file system, so if all your images derive from the same parent, that will speed up the build process and also consume less space (also during transfer!).

Edit your docker-compose.yml like this:

nginx:
  image: nginx:latest
  ports:
    - "8080:80"

php:
  image: php:7.0-fpm

nginx:

image: nginx:latest

ports:

- "8080:80"

php:

image: php:7.0-fpm

The root element is the name of the container, you can pick whatever you like. I always try to keep these short so it’s easier (less typing) when running commands against a specific container.
The image field tells docker-compose which image we want to use for our container. The ports field allows us to expose ports on the container and forward port(s) from the container to the host, so we actually connect to the container. The value "8080:80" means we’re exposing port 80 on the container and forward it to port 8080 on the host.

You should start the containers now:

$ docker-composer up -d

1	$ docker-composer up -d

Docker will pull the images from the registry, build and start them. The -d flags tells Docker to run the containers daemonized in the background. I’ll get back on that later. When it’s done, verify if they’re both up and running:

$ docker-compose ps
         Name                  Command          State               Ports             
-------------------------------------------------------------------------------------
dockertutorial_nginx_1   nginx -g daemon off;   Up      443/tcp, 0.0.0.0:8080->80/tcp 
dockertutorial_php_1     php-fpm                Up      9000/tcp

$ docker-compose ps

Name Command State Ports

-------------------------------------------------------------------------------------

dockertutorial_nginx_1 nginx -g daemon off; Up 443/tcp, 0.0.0.0:8080->80/tcp

dockertutorial_php_1 php-fpm Up 9000/tcp

Connecting to the box

We’ve forwarded port 8080 to our host, but connecting to localhost:8080 doesn’t work (you did try the link, didn’t you? :)). Because Docker runs in a virtual machine, we need to figure out its IP so we can connect it. Of course this isn’t very difficult:

$ docker-machine ip docker-tutorial
192.168.99.100

1 2	$ docker-machine ip docker-tutorial 192.168.99.100

Let’s try that IP on port 8080 and you’ll see it works: http://192.168.99.100:8080/. You’ll want to add an entry for that IP in your /etc/hosts file. Let’s pick symfony3.dev for now.

As you’ve probably discovered by now, we’re presented the default nginx page and not our shiny new Symfony application. To fix this we have to link the php container to the nginx container so they can communicate with each other. The php-fpm container needs access to our project’s php files in order to parse and serve them. Also, the nginx container requires a nginx configuration. We have to alter the Docker image and for that we need a Dockerfile.

Custom Dockerfile

The Dockerfile represents every step to be taken before the container is ready to use. Normally you would use a configuration management tool (Ansible, Puppet, Chef) to accomplish this, but in Docker you manage this via the Dockerfile.

It’s import to know that each line should contain one step. Each line creates a new layer and the number of layers is limited. One logical step per line improves the caching mechanism. For more information regarding this topic, refer to the best practices.

To configure nginx we’re going to use the nginx configuration supplied by the Symfony team. We have to copy it into the container. Create a new directory docker/nginx in the project root and add the following Dockerfile:

FROM nginx:latest

COPY symfony3.conf /etc/nginx/conf.d/symfony3.conf

FROM nginx:latest

COPY symfony3.conf /etc/nginx/conf.d/symfony3.conf

Create a symfony3.conf file in that docker/nginx directory as well and fill it with the following configuration:

server {
    server_name symfony3.dev www.symfony3.dev;
    root /app/web;

    location / {
        # try to serve file directly, fallback to app.php
        try_files $uri /app.php$is_args$args;
    }
    # DEV
    # This rule should only be placed on your development environment
    # In production, don't include this and don't deploy app_dev.php or config.php
    location ~ ^/(app_dev|config)\.php(/|$) {
        fastcgi_pass php:9000;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        # When you are using symlinks to link the document root to the
        # current version of your application, you should pass the real
        # application path instead of the path to the symlink to PHP
        # FPM.
        # Otherwise, PHP's OPcache may not properly detect changes to
        # your PHP files (see https://github.com/zendtech/ZendOptimizerPlus/issues/126
        # for more information).
        fastcgi_param  SCRIPT_FILENAME  $realpath_root$fastcgi_script_name;
        fastcgi_param DOCUMENT_ROOT $realpath_root;
    }
    # PROD
    location ~ ^/app\.php(/|$) {
        fastcgi_pass php:9000;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        # When you are using symlinks to link the document root to the
        # current version of your application, you should pass the real
        # application path instead of the path to the symlink to PHP
        # FPM.
        # Otherwise, PHP's OPcache may not properly detect changes to
        # your PHP files (see https://github.com/zendtech/ZendOptimizerPlus/issues/126
        # for more information).
        fastcgi_param  SCRIPT_FILENAME  $realpath_root$fastcgi_script_name;
        fastcgi_param DOCUMENT_ROOT $realpath_root;
        # Prevents URIs that include the front controller. This will 404:
        # http://symfony3.dev/app.php/some-path
        # Remove the internal directive to allow URIs like this
        internal;
    }
}

server {

server_name symfony3.dev www.symfony3.dev;

root /app/web;

location / {

# try to serve file directly, fallback to app.php

try_files $uri /app.php$is_args$args;

}

# DEV

# This rule should only be placed on your development environment

# In production, don't include this and don't deploy app_dev.php or config.php

location ~ ^/(app_dev|config)\.php(/|$) {

fastcgi_pass php:9000;

fastcgi_split_path_info ^(.+\.php)(/.*)$;

include fastcgi_params;

# When you are using symlinks to link the document root to the

# current version of your application, you should pass the real

# application path instead of the path to the symlink to PHP

# FPM.

# Otherwise, PHP's OPcache may not properly detect changes to

# your PHP files (see https://github.com/zendtech/ZendOptimizerPlus/issues/126

# for more information).

fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;

fastcgi_param DOCUMENT_ROOT $realpath_root;

}

# PROD

location ~ ^/app\.php(/|$) {

fastcgi_pass php:9000;

fastcgi_split_path_info ^(.+\.php)(/.*)$;

include fastcgi_params;

# When you are using symlinks to link the document root to the

# current version of your application, you should pass the real

# application path instead of the path to the symlink to PHP

# FPM.

# Otherwise, PHP's OPcache may not properly detect changes to

# your PHP files (see https://github.com/zendtech/ZendOptimizerPlus/issues/126

# for more information).

fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;

fastcgi_param DOCUMENT_ROOT $realpath_root;

# Prevents URIs that include the front controller. This will 404:

# http://symfony3.dev/app.php/some-path

# Remove the internal directive to allow URIs like this

internal;

}

In case you haven’t you should add symfony3.dev to your /etc/hosts file with the IP from docker-machine ip docker-tutorial.

Now let’s put it all together and update our docker-compose.yml accordingly:

nginx:
  build: docker/nginx
  ports:
    - "8080:80"
  links:
    - php
  volumes:
    - ./:/app

php:
  image: php:7.0-fpm
  volumes:
    - ./:/app
  working_dir: /app

nginx:

build: docker/nginx

ports:

- "8080:80"

links:

- php

volumes:

- ./:/app

php:

image: php:7.0-fpm

volumes:

- ./:/app

working_dir: /app

Take note of the changes we’ve applied: image under nginx is replaced with build: docker/nginx which refers to the directory where the Dockerfile resides. The nginx container has a links key where we link it to the php container. Both containers have a volumes key where we mount the current directory into the container under /app path. This way the container has access to the project files.

Stop all containers and build them:

$ docker-compose stop
$ docker-compose build
php uses an image, skipping
Building nginx
Step 1 : FROM nginx:latest
 ---> 5328fdfe9b8e
Step 2 : COPY symfony3.conf /etc/nginx/conf.d/symfony3.conf
 ---> Using cache
 ---> 19c62e44a1bb
Successfully built 19c62e44a1bb

$ docker-compose stop

$ docker-compose build

php uses an image, skipping

Building nginx

Step 1 : FROM nginx:latest

---> 5328fdfe9b8e

Step 2 : COPY symfony3.conf /etc/nginx/conf.d/symfony3.conf

---> Using cache

---> 19c62e44a1bb

Successfully built 19c62e44a1bb

Then start them again:

$ docker-composer up -d

1	$ docker-composer up -d

When you visit http://symfony3.dev:8080/app_dev.php in your browser, you’ll see the You are not allowed to access this file. Check app_dev.php for more information. message. Remove the access check from app_dev.php and try again.

Unfortunately another well known error pops up: Failed to write cache file “/app/var/cache/dev/classes.php”..

Permissions

In my opinion the best solution to this problem is to run the console commands and php-fpm process under the same user. Without any modifications, the console commands are run under root and the php-fpm process runs under www-data. To accomplish this we also have to use a Dockerfile for the php container.

Again, stop all containers:

$ docker-compose stop

1	$ docker-compose stop

Create a new directory php-fpm under the docker directory. Add the following Dockerfile:

FROM php:7.0-fpm

RUN useradd -ms /bin/bash vagrant

USER vagrant

WORKDIR /app

FROM php:7.0-fpm

RUN useradd -ms /bin/bash vagrant

USER vagrant

WORKDIR /app

Also, add the following php-fpm.conf file:

; This file was initially adapated from the output of: (on PHP 5.6)
;   grep -vE '^;|^ *$' /usr/local/etc/php-fpm.conf.default

[global]

error_log = /proc/stderr
daemonize = no

[www]

; if we send this to /proc/self/fd/1, it never appears
access.log = /proc/stdout

; this does the trick for changing the user
user = vagrant
group = vagrant

listen = [::]:9000

pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

clear_env = no

; Ensure worker stdout and stderr are sent to the main error log.
catch_workers_output = yes

chdir = /app/web

; This file was initially adapated from the output of: (on PHP 5.6)

; grep -vE '^;|^ *$' /usr/local/etc/php-fpm.conf.default

[global]

error_log = /proc/stderr

daemonize = no

[www]

; if we send this to /proc/self/fd/1, it never appears

access.log = /proc/stdout

; this does the trick for changing the user

user = vagrant

group = vagrant

listen = [::]:9000

pm = dynamic

pm.max_children = 5

pm.start_servers = 2

pm.min_spare_servers = 1

pm.max_spare_servers = 3

clear_env = no

; Ensure worker stdout and stderr are sent to the main error log.

catch_workers_output = yes

chdir = /app/web

Because I suck at naming new users I just use vagrant as my development user. Think of it as a tribute to vagrant :). The docker directory tree should be:

.
├── nginx
│   ├── Dockerfile
│   └── symfony3.conf
└── php-fpm
    ├── Dockerfile
    ├── php-fpm.conf

├── nginx

│ ├── Dockerfile

│ └── symfony3.conf

└── php-fpm

├── Dockerfile

├── php-fpm.conf

Now build and run the containers:

$ docker-compose build
$ docker-composer up -d

1 2	$ docker-compose build $ docker-composer up -d

If you visit http://symfony3.dev:8080/app_dev.php now, you’ll see the Symfony welcome pages smileys at you. With this “hello world” for Symfony working we end this first post.

Next post I’ll show you how to speed up things if you’re on a Mac (the default Symfony app takes ~2000 ms to load in the current situation). Also, I’ll show you the possibilities to store your data when working with containers.

Posted on February 3, 2016February 3, 2016 by ricbra 32

Starting your own local PHP usergroup

I’m a regular visitor of local usergroups. Especially if there’s an interesting talk and the location is within driving distance (which is not that hard to find in a small country as the Netherlands) me and my friends like to pay those meetups a visit. In the spring of 2015 when I was still working with Kristian Zondervan and Erik van Wingerden we often talked about things we wanted to do in our lives and Kristian talked a few times about starting a local PHP usergroup. A few conversations and a couple of football games further Erik also joined forces. As always with new things I was a bit sceptic, but I also gave it a shot. And so BredaPHP was born.

There were no guides on how to start your own meetup so we just did what we thought was good. Below you’ll find a summary of steps we took to get our meetup up and running.

Getting started

First things first: find a crew and audience.

Find co-organizers

When you want to start a local tech usergroup first try to find people who are willing to help with the organisation. Depending on how often you want to organise meetups, it does take a considerable amount of time. We are with three people and we have bimonthly meetups. We all work fulltime so we have to do the preparations in the evening or weekends. Also communication does take some time (twitter, email, meetup etc.).

Find audience

We registered our meetup on meetup.com. This is not free, but the prices are very reasonable. When registered, potential visitors will be notified by meetup.com about your new meetup. Once people join your meetup page, you’ll see others will follow quickly. We announced our first meetup a few months after the registration if I remember correctly. So it’s no problem to be inactive the first couple of months.

The first meetup

So you have some members on meetup.com? Let’s take it to the next level and prepare the first meetup.

Location

Every meetup needs a location, food and some cold beverages. Because nearly every company has one or more open PHP vacancies it shouldn’t be hard to find a location. We made a list of cool companies working with PHP in the area. We selected two companies and sent them a mail with:

who we are (BredaPHP, local usergroup for PHP developers)
what we do (organise meetups)
why we do that (meet other developers, see other companies and how they do it)
what they can offer us (a location for our meetup with drinks)
what we can offer them (a lot of potential candidates and some time for a talk about the company and how they use php)
requirements and planning for a meetup (start and endtime, schedule, requirements as beamer etc.)

Almost every company we asked so far were happy to host and sponsor a meetup.

Speaker

Speakers like to practice their new talks on meetups. Also, it turns out speakers are perfectly normal people, and when you ask a “famous” speaker most of them will come over and do a talk if they have the time. We just mailed some people with interesting talks we would like to see ourself. Also we used the Dutch PHP NL Slack group to find and invite speakers.

Depending where the meetup is hosted, we pick up speakers from the train station and bring them to the location. When the location is within walking distance, we don’t always do this.

Raffle

We also included a raffle in our first meetups. It’s relative easy to find prices:

JetBrains does sponsor licenses
O’Really does usergroup sponsering
authors often supply free copies of their work to raffle

We received some negative feedback on the raffle (visitors preferred an extra talk instead of the raffle) so we skipped in on the last meetup. Perhaps an occasional raffle will happen but we removed it from our schedule.

Next meetups

Once the first meetup is done, enjoy the positive feedback and do something with it. It’s advisable to host future meetups on recurring moments (every last Friday of the month or something like that) so people know when to expect a meetup.

Now comes the hard part: keep the meetup going. This means investing time every month. But in the end it’s all worth it. You’ll meet a lot of new people and companies. Visitors will be inspired by the meetup. After we started our meetup, two other local companies also decided to host their own development meetup, so our work didn’t go unnoticed ;).

I hope our story will help someone out there to start their own meetup!

Posted on February 1, 2016 by ricbra

How I use PHP generators to make my life easier

With the release of php 5.5 we got ourself a cool new language feature: generators. If you’re new to the concept I suggest you read the excellent blog post from Anthony Ferrara about this subject. In this post I’ll show an example of how I use generators in my every day work.

The other day I was working on a import of a large batch of vacancies from a remote system via a SOAP webservice (yes, C#.NET on the other side). The service returned a container containing the current resultset and a boolean value whether there where more results. Consider the following the classical approach without generators:

<?php

class RefreshVacanciesCommand extends ContainerAwareCommand
{
    // ...

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $service = $this->getContainer()->get('soap_service');
        $batchSize = 100;
        $startId = 0;

        do {
            $response = $service->getAllVacancies($batchSize, $startId);

            foreach ($response->GetAllVacanciesResult->Vacancies->Vacancy as $vacancy) {
                // Do actual stuff with $vacancy
            }

            $startId++;

            $more = $response->GetAllVacanciesResult->More;
        } while ($more);
    }
}

<?php

class RefreshVacanciesCommand extends ContainerAwareCommand

{

// ...

protected function execute(InputInterface $input, OutputInterface $output)

{

$service = $this->getContainer()->get('soap_service');

$batchSize = 100;

$startId = 0;

do {

$response = $service->getAllVacancies($batchSize, $startId);

foreach ($response->GetAllVacanciesResult->Vacancies->Vacancy as $vacancy) {

// Do actual stuff with $vacancy

}

$startId++;

$more = $response->GetAllVacanciesResult->More;

} while ($more);

}

There’s fundamentally not much wrong with this approach. It works. But let me show a more elegant way of solving this issue which also brings more benefits to the table as you’ll see.

The GetAllVacanciesGenerator class

Take a look at this GetAllVacanciesGenerator who’s responsibility it is to do the cumbersome fetching of the results and notice the yield keyword turning it into a Generator:

<?php

class GetAllVacanciesGenerator
{
    /**
     * @var Service
     */
    private $service;

    public function __construct(Service $service)
    {
        $this->service = $service;
    }

    /**
     * @param $batchSize
     * @param $startId
     * @return \Generator
     */
    public function getAllVacancies($batchSize, $startId)
    {
        do {
            $response = $this->service->getAllVacancies($batchSize, $startId);

            foreach ($response->GetAllVacanciesResult->Vacancies->Vacancy as $vacancy) {
                $startId = $vacancy->Id;

                yield $vacancy;
            }

            $startId++;

            $more = $response->GetAllVacanciesResult->More;
        } while ($more);
    }
}

<?php

class GetAllVacanciesGenerator

{

/**

* @var Service

private $service;

public function __construct(Service $service)

{

$this->service = $service;

}

/**

* @param $batchSize

* @param $startId

* @return \Generator

public function getAllVacancies($batchSize, $startId)

{

do {

$response = $this->service->getAllVacancies($batchSize, $startId);

foreach ($response->GetAllVacanciesResult->Vacancies->Vacancy as $vacancy) {

$startId = $vacancy->Id;

yield $vacancy;

}

$startId++;

$more = $response->GetAllVacanciesResult->More;

} while ($more);

}

The command is much cleaner now:

<?php
 
class RefreshVacanciesCommand extends ContainerAwareCommand
{
    // ...
 
    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $service = $this->getContainer()->get('soap_service');
        $batchSize = 100;
        $startId = 0;
        $generator = new GetAllVacanciesGenerator($service);
 
        foreach ($generator->getAllVacancies($batchSize, 0, null) as $vacancy) {
            // Do stuff 
        }
    }
}

<?php

class RefreshVacanciesCommand extends ContainerAwareCommand

{

// ...

protected function execute(InputInterface $input, OutputInterface $output)

{

$service = $this->getContainer()->get('soap_service');

$batchSize = 100;

$startId = 0;

$generator = new GetAllVacanciesGenerator($service);

foreach ($generator->getAllVacancies($batchSize, 0, null) as $vacancy) {

// Do stuff

}

As you can see the command is looking pretty again. I love this kind of improvements. It’s also easy to test this seperate generator class:

<?php

class GetAllVacanciesGeneratorTest extends \PHPUnit_Framework_TestCase
{
    public function test_it_should_fetch_all_vacancies()
    {
        $mock = $this->getMockBuilder('Service')
            ->disableOriginalConstructor()
            ->getMock();

        $mock->expects($this->at(0))
            ->method('getAllVacancies')
            ->with(5, 0, 'test')
            ->will($this->returnValue($this->createResponse(true, [1,2,3,4,5])));

        $mock->expects($this->at(1))
            ->method('getAllVacancies')
            ->with(5, 6, 'test')
            ->will($this->returnValue($this->createResponse(false, [6,7])));

        $mock->expects($this->exactly(2))
            ->method('getAllVacancies');

        $generator = new GetAllVacanciesGenerator($mock);

        foreach ($generator->getAllVacancies(5, 0) as $result) {
            $this->assertInstanceOf('Vacancy', $result);
        }
    }

    private function createResponse($more, $vacancyIds = [])
    {
        // Helper to ease building of fake Soap response
        ...
    }
}

<?php

class GetAllVacanciesGeneratorTest extends \PHPUnit_Framework_TestCase

{

public function test_it_should_fetch_all_vacancies()

{

$mock = $this->getMockBuilder('Service')

->disableOriginalConstructor()

->getMock();

$mock->expects($this->at(0))

->method('getAllVacancies')

->with(5, 0, 'test')

->will($this->returnValue($this->createResponse(true, [1,2,3,4,5])));

$mock->expects($this->at(1))

->method('getAllVacancies')

->with(5, 6, 'test')

->will($this->returnValue($this->createResponse(false, [6,7])));

$mock->expects($this->exactly(2))

->method('getAllVacancies');

$generator = new GetAllVacanciesGenerator($mock);

foreach ($generator->getAllVacancies(5, 0) as $result) {

$this->assertInstanceOf('Vacancy', $result);

}

private function createResponse($more, $vacancyIds = [])

{

// Helper to ease building of fake Soap response

...

}

This could also be done with implementing a Iterator but I think the generator is much more readable.

Posted on August 9, 2015 by ricbra

Symfony2 and RabbitMQ: Lessons learned

Last year we introduced RabbitMQ into our stack at Waarneembemiddeling.nl. We were in desperate need of a worker queue and after fiddling around with Gearman, Beanstalkd and RabbitMQ we made our choice: RabbitMQ it will be.

Now there’s quite some information to be found on RabbitMQ and how to use it, but a lot of things you have to find out yourself. Questions like:

what happens to messages on the queue after a service restart?
what happens to messages on the queue after a reboot?
how do we notice that a worker crashed?
what happens to my message when the consumer dies while processing?
etc.

Using RabbitMQ and Symfony2 (or php in general) is quite easy. There is a bundle for Symfony2 called OldSoundRabbitMqBundle and a php library called php-amqplib which work very well. Both are from the same author, you should probably thank him for that 🙂 .

First try: pure php consumers

We’re running a fairly common setup. Because we’ve been warned that php consumer die out every now and then, we’re using Supervisor to start new consumers when needed. There is a lot of information out there on this subject so I won’t go in there.

Despite the warnings we started with pure php consumers powered by the commands in OldSoundRabbitMqBundle. The first workers were started like this:

$ app/console rabbitmq:consumer async_event -e=prod

1	$ app/console rabbitmq:consumer async_event -e=prod

This means we’re consuming from the async_event queue without any limit to the messages. Basically this means it will run forever, or better said: until php crashes. Or worse: your consumer ends up in non-response state. Which means it doesn’t process any message any more and Supervisor thinks all is fine because you still have a running process. This happened once to our mail queue. I can assure you it’s better to prevent these kind of things.

Second try: pure php consumers with limited messages

So after the mail-gate I was searching for a quick way to make our setup more error proof. The OldSoundRabbitMqBundle supports limiting the messages to process. So I limited our workers so that they got restarted a couple of times a day:

$ app/console rabbitmq:consumer async_event -e=prod -m 10

1	$ app/console rabbitmq:consumer async_event -e=prod -m 10

After that things got running more smoothly and it took a while before we encountered new problems. While spitting trough the logs I notices some consumers produced some errors. A brief summary:

General error: 2006 MySQL server has gone away
Warning: Error while sending QUERY packet.

Because the consumer is one process that keeps running, that also means that the service container and stuff keeps existing in memory. When you’ve done some queries the database connection keeps open in the background. And if it’s quiet on our queue, it may take some time before we reach the message limit. If that time exceeds the connect_timeout of your MySQL server, you’ll run into the warnings and errors about lost connections.

Of course we should close the connection after the message is processed or could try catch for Doctrine DBAL connection exceptions or increase the connect_timeout setting but thats just denying the real problem. Running consumers with a booted Symfony2 kernel just doesn’t work so well.

A final resort could be to strip down the consumers and don’t use the Symfony2 kernel and container but we don’t like that. Our messages are most of the time serialized events which get dispatched again after the consumer picks them up. At the application level we don’t want to know wether we are in a RabbitMQ consumer or in a normal HTTP request.

Real solution: rabbitmq-cli-consumer

So it took a couple of months to learn the hard way we needed some different solution for our consumers. I found this interesting blog post about the same problem. He solved it with Java and Ruby consumers. We all learned java in college right, but I don’t like to run the memory eating jvm on our servers. The Ruby consumer unfortunately misses some good documenten for me as Ruby virgin. So I got a bit lost there.

That was the point where Go got in. Go is a kind of improved C with not real OO but a lot of cool stuff in it. I wrote a application that makes it possible to consume messages from RabbitMQ queue and pipe them into an command line application. I called it: rabbitmq-cli-consumer.

The main advantages for using rabbitmq-cli-consumer are:

no more stability issues to deal with
lightweight and fast
no need to restart your workers after a fresh deployment

We still use supervisor to start and stop the consumers because it’s the right tool for it. An example of how we start a consumer:

$ rabbitmq-cli-consumer -c example.conf -e "/home/vagrant/current/app/console wb:consumer:mail --env=prod" -V
2015/01/09 20:57:52 Setting QoS... 
2015/01/09 20:57:52 Succeeded setting QoS.
2015/01/09 20:57:52 Registering consumer... 
2015/01/09 20:57:52 Succeeded registering consumer.
2015/01/09 20:57:52 Waiting for messages...
2015/01/09 20:58:07 Processing message...
2015/01/09 20:58:13 Processed!
...

$ rabbitmq-cli-consumer -c example.conf -e "/home/vagrant/current/app/console wb:consumer:mail --env=prod" -V

2015/01/09 20:57:52 Setting QoS...

2015/01/09 20:57:52 Succeeded setting QoS.

2015/01/09 20:57:52 Registering consumer...

2015/01/09 20:57:52 Succeeded registering consumer.

2015/01/09 20:57:52 Waiting for messages...

2015/01/09 20:58:07 Processing message...

2015/01/09 20:58:13 Processed!

...

An example of a Symfony2 command we use:

namespace Wb\Bundle\CoreBundle\Command;

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class MailConsumerCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('wb:consumer:mail')
            ->addArgument('message', InputArgument::REQUIRED)
            ;
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $message = $input->getArgument('message');
        if (! $message = unserialize(base64_decode($message))) {
            throw new \InvalidArgumentException('Invalid input received');
        }

        $container = $this->getContainer();
        $mailer = $container->get('swiftmailer.mailer.mandrill');
        $transport = $container->get('swiftmailer.mailer.mandrill.transport.real');
        $logger = $container->get('rabbitmq_mail_logger');

        $logger->info(sprintf(
            '[%s] [%s] Received message from queue',
            $message->getSubject(), implode(', ', array_keys($message->getTo()))
        ));
        $spool = $mailer->getTransport()->getSpool();
        $mailer->send($message);
        $spool->flushQueue($transport);

        $logger->info(sprintf(
            '[%s] [%s] Mail send',
            $message->getSubject(), implode(', ', array_keys($message->getTo()))
        ));
    }
}

namespace Wb\Bundle\CoreBundle\Command;

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;

use Symfony\Component\Console\Input\InputArgument;

use Symfony\Component\Console\Input\InputInterface;

use Symfony\Component\Console\Output\OutputInterface;

class MailConsumerCommand extends ContainerAwareCommand

{

protected function configure()

{

$this

->setName('wb:consumer:mail')

->addArgument('message', InputArgument::REQUIRED)

;

}

protected function execute(InputInterface $input, OutputInterface $output)

{

$message = $input->getArgument('message');

if (! $message = unserialize(base64_decode($message))) {

throw new \InvalidArgumentException('Invalid input received');

}

$container = $this->getContainer();

$mailer = $container->get('swiftmailer.mailer.mandrill');

$transport = $container->get('swiftmailer.mailer.mandrill.transport.real');

$logger = $container->get('rabbitmq_mail_logger');

$logger->info(sprintf(

'[%s] [%s] Received message from queue',

$message->getSubject(), implode(', ', array_keys($message->getTo()))

));

$spool = $mailer->getTransport()->getSpool();

$mailer->send($message);

$spool->flushQueue($transport);

$logger->info(sprintf(

'[%s] [%s] Mail send',

$message->getSubject(), implode(', ', array_keys($message->getTo()))

));

}

Final tip: use the management plugin

Before even starting with RabbitMQ make sure you have the management plugin installed. It gives you a good overview about whats happening. Also you can purge queues, add users, add vhosts etc.

Posted on January 9, 2015May 7, 2015 by ricbra 13

Speed up your data migration with Spork

One of the blogs I like to keep an eye on is Kris Wallsmith his personal blog. He is a Symfony2 contributor and also author of Assetic and Buzz. Last year he wrote about a new experimental project called Spork: a wrapper around pcntl_fork to abstract away the complexities with spawning child processes with php. The article was very interesting, although I didn’t had any valid use case to try the library out. That was, until today.

It happens to be we were preparing a rather large data migration for a application with approximately 17,000 users. The legacy application stored the passwords in a unsafe way – plaintext – so we had to encrypt ’em al during the migration. Our weapon of choice was bcrypt, and using the BlowfishPasswordEncoderBundle implementing was made easy. Using bcrypt did introduce a new problem: encoding all these records would take a lot of time! That’s where Spork comes in!

Setting up the Symfony2 migration Command

If possible I wanted to fork between 8 and 15 processes to gain maximum speed. We’ll run the command on a VPS with 8 virtual cores so I want to stress the machine as much as possible ;). Unfortunately the example on GitHub as well on his blog didn’t function any more so I had to dig in just a little bit. I came up with this to get the forking working:

<?php

namespace Netvlies\AcmeMigrationBundle\Command;

use Spork\Batch\Strategy\ChunkStrategy;
use Spork\EventDispatcher\EventDispatcher;
use Spork\ProcessManager;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;

class GeneratePasswordCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('generate:password')
        ;
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $container = $this->getContainer();
        $em = $container->get('doctrine.orm.entity_manager');

        $callback = function($result) use($output) {
            $result = current($result);
            $output->writeln('Greeting from '. getmypid() .' with id '.$result['id']);
        };

        // Select all the passwords that should be encoded
        $query = $em->createQuery(
            "SELECT u.id, u.password, u.salt FROM AcmeUserBundle:User u WHERE u.isPasswordEncoded != 1 OR u.isPasswordEncoded IS NULL ORDER BY u.id"
        )->setMaxResults(10);
        $iterator = $query->iterate();

        $manager = new ProcessManager(new EventDispatcher(), true);
        $strategy = new ChunkStrategy();
        $manager->process($iterator, $callback, $strategy);
    }
}

<?php

namespace Netvlies\AcmeMigrationBundle\Command;

use Spork\Batch\Strategy\ChunkStrategy;

use Spork\EventDispatcher\EventDispatcher;

use Spork\ProcessManager;

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;

use Symfony\Component\Console\Input\InputArgument;

use Symfony\Component\Console\Input\InputInterface;

use Symfony\Component\Console\Input\InputOption;

class GeneratePasswordCommand extends ContainerAwareCommand

{

protected function configure()

{

$this

->setName('generate:password')

;

}

protected function execute(InputInterface $input, OutputInterface $output)

{

$container = $this->getContainer();

$em = $container->get('doctrine.orm.entity_manager');

$callback = function($result) use($output) {

$result = current($result);

$output->writeln('Greeting from '. getmypid() .' with id '.$result['id']);

};

// Select all the passwords that should be encoded

$query = $em->createQuery(

"SELECT u.id, u.password, u.salt FROM AcmeUserBundle:User u WHERE u.isPasswordEncoded != 1 OR u.isPasswordEncoded IS NULL ORDER BY u.id"

)->setMaxResults(10);

$iterator = $query->iterate();

$manager = new ProcessManager(new EventDispatcher(), true);

$strategy = new ChunkStrategy();

$manager->process($iterator, $callback, $strategy);

}

The command generates the following output:

$ app/console wb:generate:password
Greeting from 3743 with id 1
Greeting from 3743 with id 2
Greeting from 3743 with id 3
Greeting from 3744 with id 4
Greeting from 3744 with id 5
Greeting from 3744 with id 6
Greeting from 3745 with id 7
Greeting from 3745 with id 8
Greeting from 3745 with id 9
Greeting from 3746 with id 10

$ app/console wb:generate:password

Greeting from 3743 with id 1

Greeting from 3743 with id 2

Greeting from 3743 with id 3

Greeting from 3744 with id 4

Greeting from 3744 with id 5

Greeting from 3744 with id 6

Greeting from 3745 with id 7

Greeting from 3745 with id 8

Greeting from 3745 with id 9

Greeting from 3746 with id 10

Make it a little bit more dynamic

To be really useful I’ve added some parameters so we can control the behavior a little more. As I mentioned before I wanted to control the amount forks so I added a option to control this. This value needs to be passed on to the constructor of the ChunkStrategy:

<?php
 
namespace Netvlies\AcmeMigrationBundle\Command;
 
...
 
class GeneratePasswordCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('generate:password')
            ->addOption('forks', 'f', InputOption::VALUE_REQUIRED, 'How many childs to be spawned', 4)
        ;
    }
 
    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $forks = (int) $input->getOption('forks');
 
        ...

        $manager = new ProcessManager(new EventDispatcher(), true);
        $strategy = new ChunkStrategy($forks);
        $manager->process($iterator, $callback, $strategy);
    }
}

<?php

namespace Netvlies\AcmeMigrationBundle\Command;

...

class GeneratePasswordCommand extends ContainerAwareCommand

{

protected function configure()

{

$this

->setName('generate:password')

->addOption('forks', 'f', InputOption::VALUE_REQUIRED, 'How many childs to be spawned', 4)

;

}

protected function execute(InputInterface $input, OutputInterface $output)

{

$forks = (int) $input->getOption('forks');

...

$manager = new ProcessManager(new EventDispatcher(), true);

$strategy = new ChunkStrategy($forks);

$manager->process($iterator, $callback, $strategy);

}

I also added a max parameter so we can run some tests on a small set of users, instead of the whole database. When set I pass it on to the setMaxResults method of the $query object.

Storing the results in MySQL: Beware!

In Symfony2 projects storing and reading data from the database is pretty straight forward using Doctrine2. However when you start forking your PHP process keep in mind the following:

all the forks share the same database connection;
when the first fork exits, it will also close the database connection;
database operations in running forks will yield: General error: 2006 MySQL server has gone away

This is a known problem. In order to fix this problem I create and close a new connection in each fork:

<?php

use Doctrine\DBAL\DriverManager;
 
class GeneratePasswordCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('generate:password')
            ->addOption('forks', 'f', InputOption::VALUE_REQUIRED, 'How many childs to be spawned', 4)
            ->addOption('max', 'm', InputOption::VALUE_OPTIONAL, 'Maximum records to process')
        ;
    }
 
    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $forks = (int) $input->getOption('forks');
        $container = $this->getContainer();
        $em = $container->get('doctrine.orm.entity_manager');
        $encoder = $container->get('security.encoder.blowfish');
        $params = array(
            'dbname'    => $container->getParameter('database_name'),
            'user'      => $container->getParameter('database_user'),
            'password'  => $container->getParameter('database_password'),
            'host'      => $container->getParameter('database_host'),
            'driver'    => $container->getParameter('database_driver'),
        );
 
        $callback = function($result) use ($encoder, $output, $params) {
            // Create connection
            $conn = DriverManager::getConnection($params);
            $conn->connect();
            $result = current($result);
            $encodedPassword = $encoder->encodePassword($result['password'], $result['salt']);
            // Store result
            $stmt = $conn->prepare("UPDATE User u SET u.password = ?, u.isPasswordEncoded = 1 WHERE u.id = ?");
            $stmt->bindValue(1, $encodedPassword);
            $stmt->bindValue(2, $result['id']);
            $stmt->execute();
            // Close connection
            $conn->close();
            $output->writeln(sprintf('Encoded password for id: %d is %s', $result['id'], $encodedPassword));
        };
 
        $query = $em->createQuery(
            "SELECT u.id, u.password, u.salt FROM AcmeUserBundle:User u WHERE u.isPasswordEncoded != 1 OR u.isPasswordEncoded IS NULL ORDER BY u.id"
        );
        if ($max = $input->getOption('max')) {
            $query->setMaxResults((int) $max);
        }
        $iterator = $query->iterate();
 
        $manager = new ProcessManager(new EventDispatcher(), true);
        $strategy = new ChunkStrategy($forks);
        $manager->process($iterator, $callback, $strategy);
    }
}

<?php

use Doctrine\DBAL\DriverManager;

class GeneratePasswordCommand extends ContainerAwareCommand

{

protected function configure()

{

$this

->setName('generate:password')

->addOption('forks', 'f', InputOption::VALUE_REQUIRED, 'How many childs to be spawned', 4)

->addOption('max', 'm', InputOption::VALUE_OPTIONAL, 'Maximum records to process')

;

}

protected function execute(InputInterface $input, OutputInterface $output)

{

$forks = (int) $input->getOption('forks');

$container = $this->getContainer();

$em = $container->get('doctrine.orm.entity_manager');

$encoder = $container->get('security.encoder.blowfish');

$params = array(

'dbname' => $container->getParameter('database_name'),

'user' => $container->getParameter('database_user'),

'password' => $container->getParameter('database_password'),

'host' => $container->getParameter('database_host'),

'driver' => $container->getParameter('database_driver'),

);

$callback = function($result) use ($encoder, $output, $params) {

// Create connection

$conn = DriverManager::getConnection($params);

$conn->connect();

$result = current($result);

$encodedPassword = $encoder->encodePassword($result['password'], $result['salt']);

// Store result

$stmt = $conn->prepare("UPDATE User u SET u.password = ?, u.isPasswordEncoded = 1 WHERE u.id = ?");

$stmt->bindValue(1, $encodedPassword);

$stmt->bindValue(2, $result['id']);

$stmt->execute();

// Close connection

$conn->close();

$output->writeln(sprintf('Encoded password for id: %d is %s', $result['id'], $encodedPassword));

};

$query = $em->createQuery(

"SELECT u.id, u.password, u.salt FROM AcmeUserBundle:User u WHERE u.isPasswordEncoded != 1 OR u.isPasswordEncoded IS NULL ORDER BY u.id"

);

if ($max = $input->getOption('max')) {

$query->setMaxResults((int) $max);

}

$iterator = $query->iterate();

$manager = new ProcessManager(new EventDispatcher(), true);

$strategy = new ChunkStrategy($forks);

$manager->process($iterator, $callback, $strategy);

}

That’s basically it. Running this command on a VPS comparable with c1.xlarge Amazone EC2 server did speed up things a lot. So if you’re also working on a import job like this which can be split up in separate tasks you know what to do… Give Spork a try! It’s really easy, I promise.

UPDATE 2013-03-19
As stated in the comments by Kris, you should close the connection just before forking. Example of how to do this:

<?php
 
namespace Netvlies\AcmeMigrationBundle\Command;
 
...
use Spork\EventDispatcher\Events;
 
class GeneratePasswordCommand extends ContainerAwareCommand
{
    ...

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        ...
        // Get connection in parent
        $parentConnection = $em->getConnection();
        $dispatcher = new EventDispatcher();
        $dispatcher->addListener(Events::PRE_FORK, function() use ($parentConnection) {
            $parentConnection->close();
        });
        $manager = new ProcessManager($dispatcher, true);
        $strategy = new ChunkStrategy($forks);
        $manager->process($iterator, $callback, $strategy);
    }
}

<?php

namespace Netvlies\AcmeMigrationBundle\Command;

...

use Spork\EventDispatcher\Events;

class GeneratePasswordCommand extends ContainerAwareCommand

{

...

protected function execute(InputInterface $input, OutputInterface $output)

{

...

// Get connection in parent

$parentConnection = $em->getConnection();

$dispatcher = new EventDispatcher();

$dispatcher->addListener(Events::PRE_FORK, function() use ($parentConnection) {

$parentConnection->close();

});

$manager = new ProcessManager($dispatcher, true);

$strategy = new ChunkStrategy($forks);

$manager->process($iterator, $callback, $strategy);

}

Posted on March 17, 2013April 28, 2015 by ricbra 6

Symfony2 authentication provider: authenticate against webservice

The past few days I have really be struggeling with the Symfony2 security component. It is the most complex component of Symfony2 if you ask me! On the symfony.com website there is a pretty neat cookbook article about creating a custom authentication provider. Despite the fact that it covers the subject pretty well, it lacks support for form-based authentication use cases. In the current Symfony2 project I’m working on, we’re dealing with a web service that we need to authenticate against. So the cookbook article was nothing more then a good introduction unfortunately.

Using DaoAuthenticationProvider as example

Since we don’t want to reinvent the wheel, a good place to start is by investigating the providers that are in the Symfony2 core. The DaoAuthenticationProvider is a very good example, and used by the default form login. We are going to add a few pieces of code, so we can use the listener and configuration settings. The only thing we want to change are the authentication itself and the user provider. If you take a look at the link above, you will see the only thing we need to change is the checkAuthentication method. But, a few more steps are needed in order to make things function correctly. Let’s begin! 🙂

We also need a UserProvider!

First things first: we need a custom user provider. The task of the user provider is load the user from a source so the authentication process can continue. Because a user can already be registered at the webservice a traditional database user provider won’t work. We need to create a local record for every user that registers or logs in and doesn’t have an account. So basically the user provider is only responsible for loading and creating a user record. In this example I save the user immediately when there is no record; probably you want to do this after authenticating.

The code for the use provider looks like this:

<?php

namespace Acme\DemoBundle\Security\Core\User;

use Symfony\Component\Security\Core\User\UserProviderInterface;
use Symfony\Component\Security\Core\User\UserInterface;
use Symfony\Component\Security\Core\Exception\UsernameNotFoundException;
use Symfony\Component\Security\Core\Exception\UnsupportedUserException;
use Acme\DemoBundle\Service\Service;
use Acme\DemoBundle\Entity\User;
use Doctrine\ORM\EntityManager;

class WebserviceUserProvider implements UserProviderInterface
{
    private $service;
    private $em;

    public function __construct(Service $service, EntityManager $em)
    {
        $this->service  = $service;
        $this->em       = $em;
    }

    public function loadUserByUsername($username)
    {
        // Do we have a local record?
        if ($user = $this->findUserBy(array('email' => $username))) {
            return $user;
        }

        // Try service
        if ($record = $this->service->getUser($username)) {
            // Set some fields
            $user = new User();
            $user->setUsername($username);
            return $user;
        }

        throw new UsernameNotFoundException(sprintf('No record found for user %s', $username));
    }

    public function refreshUser(UserInterface $user)
    {
        return $this->loadUserByUsername($user->getUsername());
    }

    public function supportsClass($class)
    {
        return $class === 'Acme\DemoBundle\Entity\User';
    }

    protected function findUserBy(array $criteria)
    {
        $repository = $this->em->getRepository('Acme\DemoBundle\Entity\User');
        return $repository->findOneBy($criteria);
    }
}

<?php

namespace Acme\DemoBundle\Security\Core\User;

use Symfony\Component\Security\Core\User\UserProviderInterface;

use Symfony\Component\Security\Core\User\UserInterface;

use Symfony\Component\Security\Core\Exception\UsernameNotFoundException;

use Symfony\Component\Security\Core\Exception\UnsupportedUserException;

use Acme\DemoBundle\Service\Service;

use Acme\DemoBundle\Entity\User;

use Doctrine\ORM\EntityManager;

class WebserviceUserProvider implements UserProviderInterface

{

private $service;

private $em;

public function __construct(Service $service, EntityManager $em)

{

$this->service = $service;

$this->em = $em;

}

public function loadUserByUsername($username)

{

// Do we have a local record?

if ($user = $this->findUserBy(array('email' => $username))) {

return $user;

}

// Try service

if ($record = $this->service->getUser($username)) {

// Set some fields

$user = new User();

$user->setUsername($username);

return $user;

}

throw new UsernameNotFoundException(sprintf('No record found for user %s', $username));

}

public function refreshUser(UserInterface $user)

{

return $this->loadUserByUsername($user->getUsername());

}

public function supportsClass($class)

{

return $class === 'Acme\DemoBundle\Entity\User';

}

protected function findUserBy(array $criteria)

{

$repository = $this->em->getRepository('Acme\DemoBundle\Entity\User');

return $repository->findOneBy($criteria);

}

We add it to our services configuration in app/config/services.yml:

<container xmlns="http://symfony.com/schema/dic/services"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://symfony.com/schema/dic/services http://symfony.com/schema/dic/services/services-1.0.xsd">

    <parameters>
        <parameter key="security.user.provider.acme.service.class">Acme\DemoBundle\Security\Core\User\WebserviceUserProvider</parameter>
        <parameter key="acme.service.class">Acme\DemoBundle\Service\Service</parameter>
    </parameters>

    <services>
        <service id="acme_demo_webservice" class="%acme.service.class%">
        </service>
        <service id="acme_demo_user_provider" class="%security.user.provider.acme.service.class%">
            <argument type="service" id="acme_demo_webservice" />
            <argument type="service" id="doctrine.orm.entity_manager" />
        </service>
    </services>
</container>

<container xmlns="http://symfony.com/schema/dic/services"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://symfony.com/schema/dic/services http://symfony.com/schema/dic/services/services-1.0.xsd">

<parameter key="security.user.provider.acme.service.class">Acme\DemoBundle\Security\Core\User\WebserviceUserProvider</parameter>

<parameter key="acme.service.class">Acme\DemoBundle\Service\Service</parameter>

</parameters>

</service>

</service>

</services>

</container>

Creating the AuthenticationProvider

As I said earlier we are going to base our provider on the DaoAuthenticationProvider. In my bundle I created a new class called ServiceAuthenticationProvider. Like our example we are extending the abstract UserAuthenticationProvider. Besides the checkAuthentication method we also must implement the retrieveUser method. We inject the service through the constructor, so the class looks like this:

<?php
namespace Acme\DemoBundle\Security\Core\Authentication\Provider;

use Symfony\Component\Security\Core\Encoder\EncoderFactoryInterface;
use Symfony\Component\Security\Core\User\UserProviderInterface;
use Symfony\Component\Security\Core\User\UserCheckerInterface;
use Symfony\Component\Security\Core\User\UserInterface;
use Symfony\Component\Security\Core\Exception\UsernameNotFoundException;
use Symfony\Component\Security\Core\Exception\AuthenticationServiceException;
use Symfony\Component\Security\Core\Exception\BadCredentialsException;
use Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken;
use Symfony\Component\Security\Core\Authentication\Provider\UserAuthenticationProvider;
use Acme\DemoBundle\Service\Service;

class EpsAuthenticationProvider extends UserAuthenticationProvider
{
    private $encoderFactory;
    private $userProvider;
    private $service;

    /**
     * @param Service $service
     * @param \Symfony\Component\Security\Core\User\UserProviderInterface $userProvider
     * @param UserCheckerInterface $userChecker
     * @param $providerKey
     * @param EncoderFactoryInterface $encoderFactory
     * @param bool $hideUserNotFoundExceptions
     */
    public function __construct(
        Service $service,
        UserProviderInterface $userProvider,
        UserCheckerInterface $userChecker,
        $providerKey,
        EncoderFactoryInterface $encoderFactory,
        $hideUserNotFoundExceptions = true
    ) {
        parent::__construct($userChecker, $providerKey, $hideUserNotFoundExceptions);
        $this->encoderFactory = $encoderFactory;
        $this->userProvider = $userProvider;
        $this->service = $service;
    }

    /**
     * {@inheritdoc}
     */
    protected function checkAuthentication(UserInterface $user, UsernamePasswordToken $token)
    {
        $currentUser = $token->getUser();

        if ($currentUser instanceof UserInterface) {
            if ($currentUser->getPassword() !== $user->getPassword()) {
                throw new BadCredentialsException('The credentials were changed from another session.');
            }
        } else {
            if (!$presentedPassword = $token->getCredentials()) {
                throw new BadCredentialsException('The presented password cannot be empty.');
            }

            if (!$this->service->authenticate($token->getUser(), $presentedPassword)) {
                throw new BadCredentialsException('The presented password is invalid.');
            }
        }
    }

    /**
     * {@inheritdoc}
     */
    protected function retrieveUser($username, UsernamePasswordToken $token)
    {
        $user = $token->getUser();
        if ($user instanceof UserInterface) {
            return $user;
        }

        try {
            $user = $this->userProvider->loadUserByUsername($username);

            if (!$user instanceof UserInterface) {
                throw new AuthenticationServiceException('The user provider must return a UserInterface object.');
            }

            return $user;
        } catch (UsernameNotFoundException $notFound) {
            throw $notFound;
        } catch (\Exception $repositoryProblem) {
            throw new AuthenticationServiceException($repositoryProblem->getMessage(), $token, 0, $repositoryProblem);
        }
    }
}

<?php

namespace Acme\DemoBundle\Security\Core\Authentication\Provider;

use Symfony\Component\Security\Core\Encoder\EncoderFactoryInterface;

use Symfony\Component\Security\Core\User\UserProviderInterface;

use Symfony\Component\Security\Core\User\UserCheckerInterface;

use Symfony\Component\Security\Core\User\UserInterface;

use Symfony\Component\Security\Core\Exception\UsernameNotFoundException;

use Symfony\Component\Security\Core\Exception\AuthenticationServiceException;

use Symfony\Component\Security\Core\Exception\BadCredentialsException;

use Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken;

use Symfony\Component\Security\Core\Authentication\Provider\UserAuthenticationProvider;

use Acme\DemoBundle\Service\Service;

class EpsAuthenticationProvider extends UserAuthenticationProvider

{

private $encoderFactory;

private $userProvider;

private $service;

/**

* @param Service $service

* @param \Symfony\Component\Security\Core\User\UserProviderInterface $userProvider

* @param UserCheckerInterface $userChecker

* @param $providerKey

* @param EncoderFactoryInterface $encoderFactory

* @param bool $hideUserNotFoundExceptions

public function __construct(

Service $service,

UserProviderInterface $userProvider,

UserCheckerInterface $userChecker,

$providerKey,

EncoderFactoryInterface $encoderFactory,

$hideUserNotFoundExceptions = true

) {

parent::__construct($userChecker, $providerKey, $hideUserNotFoundExceptions);

$this->encoderFactory = $encoderFactory;

$this->userProvider = $userProvider;

$this->service = $service;

}

/**

* {@inheritdoc}

protected function checkAuthentication(UserInterface $user, UsernamePasswordToken $token)

{

$currentUser = $token->getUser();

if ($currentUser instanceof UserInterface) {

if ($currentUser->getPassword() !== $user->getPassword()) {

throw new BadCredentialsException('The credentials were changed from another session.');

}

} else {

if (!$presentedPassword = $token->getCredentials()) {

throw new BadCredentialsException('The presented password cannot be empty.');

}

if (!$this->service->authenticate($token->getUser(), $presentedPassword)) {

throw new BadCredentialsException('The presented password is invalid.');

}

/**

* {@inheritdoc}

protected function retrieveUser($username, UsernamePasswordToken $token)

{

$user = $token->getUser();

if ($user instanceof UserInterface) {

return $user;

}

try {

$user = $this->userProvider->loadUserByUsername($username);

if (!$user instanceof UserInterface) {

throw new AuthenticationServiceException('The user provider must return a UserInterface object.');

}

return $user;

} catch (UsernameNotFoundException $notFound) {

throw $notFound;

} catch (\Exception $repositoryProblem) {

throw new AuthenticationServiceException($repositoryProblem->getMessage(), $token, 0, $repositoryProblem);

}

Note the call to $this->service->authenticate where the magic happens. The retrieveUser method receives a User instance from our user provider. Although this is not really clear in the code above, it will be after configuration in the service container. We use the configuration from the Symfony core and adjust it to our needs:

<service id="security.authentication_provider.acme_demo_webservice" class="%security.authentication.provider.acme_service.class%" abstract="true" public="false">
    <argument type="service" id="acme_demo_webservice" />
    <argument /> <!– User Provider –>
    <argument type="service" id="security.user_checker" />
    <argument /> <!– Provider-shared Key –>
    <argument type="service" id="security.encoder_factory" />
    <argument>%security.authentication.hide_user_not_found%</argument>
</service>

<argument>%security.authentication.hide_user_not_found%</argument>

</service>

Please note the empty arguments. Look a bit strange, huh? These will be magically filled when the container is build by our Factory! This is a bit tricky, and the cookbook explains pretty wel, so I suggest to take a look there. We are extending the FormLoginFactory because we want to change it bit:

<?php

namespace Acme\DemoBundle\DependencyInjection\Factory;

use Symfony\Component\DependencyInjection\DefinitionDecorator;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\DependencyInjection\Reference;
use Symfony\Bundle\SecurityBundle\DependencyInjection\Security\Factory\FormLoginFactory;

class SecurityFactory extends FormLoginFactory
{
    public function getKey()
    {
        return 'webservice-login';
    }

    protected function getListenerId()
    {
        return 'security.authentication.listener.form';
    }

    protected function createAuthProvider(ContainerBuilder $container, $id, $config, $userProviderId)
    {
        $provider = 'security.authentication_provider.acme_demo_webservice.'.$id;
        $container
            ->setDefinition($provider, new DefinitionDecorator('security.authentication_provider.acme_demo_webservice'))
            ->replaceArgument(1, new Reference($userProviderId))
            ->replaceArgument(3, $id);
 
        return $provider;
    }
}

<?php

namespace Acme\DemoBundle\DependencyInjection\Factory;

use Symfony\Component\DependencyInjection\DefinitionDecorator;

use Symfony\Component\DependencyInjection\ContainerBuilder;

use Symfony\Component\DependencyInjection\Reference;

use Symfony\Bundle\SecurityBundle\DependencyInjection\Security\Factory\FormLoginFactory;

class SecurityFactory extends FormLoginFactory

{

public function getKey()

{

return 'webservice-login';

}

protected function getListenerId()

{

return 'security.authentication.listener.form';

}

protected function createAuthProvider(ContainerBuilder $container, $id, $config, $userProviderId)

{

$provider = 'security.authentication_provider.acme_demo_webservice.'.$id;

$container

->setDefinition($provider, new DefinitionDecorator('security.authentication_provider.acme_demo_webservice'))

->replaceArgument(1, new Reference($userProviderId))

->replaceArgument(3, $id);

return $provider;

}

Add the builder in the Acme\DemoBundle\AcmeDemoBundle.php file:

<?php

namespace Acme\DemoBundle;

use Symfony\Component\HttpKernel\Bundle\Bundle;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Acme\DemoBundle\DependencyInjection\Factory\SecurityFactory;

class AcmeDemoBundle extends Bundle
{
    public function build(ContainerBuilder $container)
    {
        $extension = $container->getExtension('security');
        $extension->addSecurityListenerFactory(new SecurityFactory());
    }
}

<?php

namespace Acme\DemoBundle;

use Symfony\Component\HttpKernel\Bundle\Bundle;

use Symfony\Component\DependencyInjection\ContainerBuilder;

use Acme\DemoBundle\DependencyInjection\Factory\SecurityFactory;

class AcmeDemoBundle extends Bundle

{

public function build(ContainerBuilder $container)

{

$extension = $container->getExtension('security');

$extension->addSecurityListenerFactory(new SecurityFactory());

}

Finally, change your security config:

jms_security_extra:
    secure_all_services: false
    expressions: true

security:
    encoders:
        Symfony\Component\Security\Core\User\User: plaintext

    role_hierarchy:
        ROLE_ADMIN:       ROLE_USER
        ROLE_SUPER_ADMIN: [ROLE_USER, ROLE_ADMIN, ROLE_ALLOWED_TO_SWITCH]

    providers:
        acme_provider:
            id: acme_demo_user_provider

    firewalls:
        dev:
            pattern:  ^/(_(profiler|wdt)|css|images|js)/
            security: false

        login:
            pattern:  ^/demo/secured/login$
            security: false

        secured_area:
            pattern:    ^/demo/secured/
            webservice-login:
                check_path: /demo/secured/login_check
                login_path: /demo/secured/login
                provider: acme_provider
            logout:
                path:   /demo/secured/logout
                target: /demo/


    access_control:

jms_security_extra:

secure_all_services: false

expressions: true

security:

encoders:

Symfony\Component\Security\Core\User\User: plaintext

role_hierarchy:

ROLE_ADMIN: ROLE_USER

ROLE_SUPER_ADMIN: [ROLE_USER, ROLE_ADMIN, ROLE_ALLOWED_TO_SWITCH]

providers:

acme_provider:

id: acme_demo_user_provider

firewalls:

dev:

pattern: ^/(_(profiler|wdt)|css|images|js)/

security: false

pattern: ^/demo/secured/login$

security: false

secured_area:

pattern: ^/demo/secured/

webservice-login:

check_path: /demo/secured/login_check

login_path: /demo/secured/login

provider: acme_provider

logout:

path: /demo/secured/logout

target: /demo/

access_control:

The webservice-login key activates our authentication provider. The user provider is defined under providers as acme_provider with the corresponding service id.
I used the AcmeDemo bundle from symfony-standard repository, so you could just copy paste most of my code to see everything in action! Only thing you need to provide yourself is a dummy webservice.

Happy coding!

Posted on June 19, 2012April 28, 2015 by ricbra 39

Ricbra's Blog

Loving the web since 2001

Ricbra's Blog

Tag: PHP

Developing Symfony applications with Docker series part I: Getting started

Installation

Create Linux virtual box

Create Symfony project

Configuring docker-compose

Install php and nginx

Connecting to the box

Custom Dockerfile

Permissions

Starting your own local PHP usergroup

Getting started

Find co-organizers

Find audience

The first meetup

Location

Speaker

Raffle

Next meetups

How I use PHP generators to make my life easier

The GetAllVacanciesGenerator class

Symfony2 and RabbitMQ: Lessons learned

First try: pure php consumers

Second try: pure php consumers with limited messages

Real solution: rabbitmq-cli-consumer

Final tip: use the management plugin

Speed up your data migration with Spork

Setting up the Symfony2 migration Command

Make it a little bit more dynamic

Storing the results in MySQL: Beware!

Symfony2 authentication provider: authenticate against webservice

Using DaoAuthenticationProvider as example

We also need a UserProvider!

Creating the AuthenticationProvider