How I use PHP generators to make my life easier

With the release of php 5.5 we got ourself a cool new language feature: generators. If you’re new to the concept I suggest you read the excellent blog post from Anthony Ferrara about this subject. In this post I’ll show an example of how I use generators in my every day work.

The other day I was working on a import of a large batch of vacancies from a remote system via a SOAP webservice (yes, C#.NET on the other side). The service returned a container containing the current resultset and a boolean value whether there where more results. Consider the following the classical approach without generators:

<?php

class RefreshVacanciesCommand extends ContainerAwareCommand
{
    // ...

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $service = $this->getContainer()->get('soap_service');
        $batchSize = 100;
        $startId = 0;

        do {
            $response = $service->getAllVacancies($batchSize, $startId);

            foreach ($response->GetAllVacanciesResult->Vacancies->Vacancy as $vacancy) {
                // Do actual stuff with $vacancy
            }

            $startId++;

            $more = $response->GetAllVacanciesResult->More;
        } while ($more);
    }
}

<?php

class RefreshVacanciesCommand extends ContainerAwareCommand

{

// ...

protected function execute(InputInterface $input, OutputInterface $output)

{

$service = $this->getContainer()->get('soap_service');

$batchSize = 100;

$startId = 0;

do {

$response = $service->getAllVacancies($batchSize, $startId);

foreach ($response->GetAllVacanciesResult->Vacancies->Vacancy as $vacancy) {

// Do actual stuff with $vacancy

}

$startId++;

$more = $response->GetAllVacanciesResult->More;

} while ($more);

}

There’s fundamentally not much wrong with this approach. It works. But let me show a more elegant way of solving this issue which also brings more benefits to the table as you’ll see.

The GetAllVacanciesGenerator class

Take a look at this GetAllVacanciesGenerator who’s responsibility it is to do the cumbersome fetching of the results and notice the yield keyword turning it into a Generator:

<?php

class GetAllVacanciesGenerator
{
    /**
     * @var Service
     */
    private $service;

    public function __construct(Service $service)
    {
        $this->service = $service;
    }

    /**
     * @param $batchSize
     * @param $startId
     * @return \Generator
     */
    public function getAllVacancies($batchSize, $startId)
    {
        do {
            $response = $this->service->getAllVacancies($batchSize, $startId);

            foreach ($response->GetAllVacanciesResult->Vacancies->Vacancy as $vacancy) {
                $startId = $vacancy->Id;

                yield $vacancy;
            }

            $startId++;

            $more = $response->GetAllVacanciesResult->More;
        } while ($more);
    }
}

<?php

class GetAllVacanciesGenerator

{

/**

* @var Service

private $service;

public function __construct(Service $service)

{

$this->service = $service;

}

/**

* @param $batchSize

* @param $startId

* @return \Generator

public function getAllVacancies($batchSize, $startId)

{

do {

$response = $this->service->getAllVacancies($batchSize, $startId);

foreach ($response->GetAllVacanciesResult->Vacancies->Vacancy as $vacancy) {

$startId = $vacancy->Id;

yield $vacancy;

}

$startId++;

$more = $response->GetAllVacanciesResult->More;

} while ($more);

}

The command is much cleaner now:

<?php
 
class RefreshVacanciesCommand extends ContainerAwareCommand
{
    // ...
 
    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $service = $this->getContainer()->get('soap_service');
        $batchSize = 100;
        $startId = 0;
        $generator = new GetAllVacanciesGenerator($service);
 
        foreach ($generator->getAllVacancies($batchSize, 0, null) as $vacancy) {
            // Do stuff 
        }
    }
}

<?php

class RefreshVacanciesCommand extends ContainerAwareCommand

{

// ...

protected function execute(InputInterface $input, OutputInterface $output)

{

$service = $this->getContainer()->get('soap_service');

$batchSize = 100;

$startId = 0;

$generator = new GetAllVacanciesGenerator($service);

foreach ($generator->getAllVacancies($batchSize, 0, null) as $vacancy) {

// Do stuff

}

As you can see the command is looking pretty again. I love this kind of improvements. It’s also easy to test this seperate generator class:

<?php

class GetAllVacanciesGeneratorTest extends \PHPUnit_Framework_TestCase
{
    public function test_it_should_fetch_all_vacancies()
    {
        $mock = $this->getMockBuilder('Service')
            ->disableOriginalConstructor()
            ->getMock();

        $mock->expects($this->at(0))
            ->method('getAllVacancies')
            ->with(5, 0, 'test')
            ->will($this->returnValue($this->createResponse(true, [1,2,3,4,5])));

        $mock->expects($this->at(1))
            ->method('getAllVacancies')
            ->with(5, 6, 'test')
            ->will($this->returnValue($this->createResponse(false, [6,7])));

        $mock->expects($this->exactly(2))
            ->method('getAllVacancies');

        $generator = new GetAllVacanciesGenerator($mock);

        foreach ($generator->getAllVacancies(5, 0) as $result) {
            $this->assertInstanceOf('Vacancy', $result);
        }
    }

    private function createResponse($more, $vacancyIds = [])
    {
        // Helper to ease building of fake Soap response
        ...
    }
}

<?php

class GetAllVacanciesGeneratorTest extends \PHPUnit_Framework_TestCase

{

public function test_it_should_fetch_all_vacancies()

{

$mock = $this->getMockBuilder('Service')

->disableOriginalConstructor()

->getMock();

$mock->expects($this->at(0))

->method('getAllVacancies')

->with(5, 0, 'test')

->will($this->returnValue($this->createResponse(true, [1,2,3,4,5])));

$mock->expects($this->at(1))

->method('getAllVacancies')

->with(5, 6, 'test')

->will($this->returnValue($this->createResponse(false, [6,7])));

$mock->expects($this->exactly(2))

->method('getAllVacancies');

$generator = new GetAllVacanciesGenerator($mock);

foreach ($generator->getAllVacancies(5, 0) as $result) {

$this->assertInstanceOf('Vacancy', $result);

}

private function createResponse($more, $vacancyIds = [])

{

// Helper to ease building of fake Soap response

...

}

This could also be done with implementing a Iterator but I think the generator is much more readable.

Posted on August 9, 2015 by ricbra

Creating a custom Doctrine DBAL type the right way

Today one of my colleague’s was debugging a strange issue with Doctrine’s schema validation tool which caused our test setup to fail (we’re running app/console doctrine:schema:validate as part of our CI process). The output was the same every time:

$ app/console doctrine:schema:validate
[Mapping]  OK - The mapping files are correct.
[Database] FAIL - The database schema is not in sync with the current mapping file.

$ app/console doctrine:schema:validate

[Mapping] OK - The mapping files are correct.

[Database] FAIL - The database schema is not in sync with the current mapping file.

We quickly discovered the issue was caused by the custom UUID Doctrine DBAL type we introduced lately. This type was based on some gist we found on the web:

<?php

namespace BuboBox\Doctrine2\DBAL\Types;

use Doctrine\DBAL\Types\Type;
use Doctrine\DBAL\Platforms\AbstractPlatform;

/**
 * Type that maps a PHP array to a clob SQL type.
 *
 * @since 2.0
 */
class UuidType extends Type
{

    const BINARY = 'binary';

    public function getSqlDeclaration(array $fieldDeclaration, AbstractPlatform $platform)
    {
        return sprintf('BINARY(%d)', $fieldDeclaration['length']);
    }

    public function getName()
    {       
        return self::BINARY;
    }   

    public function convertToPhpValue($value, AbstractPlatform $platform)
    {
        if ($value !== null) {
            $value= unpack('H*', $value);
            $hash = array_shift($value);
            $uuid = substr($hash,  0,  8) . '-' . substr($hash,  8,  4) . '-' . substr($hash, 12,  4) . '-' . substr($hash, 16,  4) . '-' . substr($hash, 20, 12);
            return $uuid;
        }
    }

    public function convertToDatabaseValue($value, AbstractPlatform $platform)
    {
        if ($value !== null) {
            return pack('H*', str_replace('-', '',$value));
        }
    }

}

<?php

namespace BuboBox\Doctrine2\DBAL\Types;

use Doctrine\DBAL\Types\Type;

use Doctrine\DBAL\Platforms\AbstractPlatform;

/**

* Type that maps a PHP array to a clob SQL type.

* @since 2.0

class UuidType extends Type

{

const BINARY = 'binary';

public function getSqlDeclaration(array $fieldDeclaration, AbstractPlatform $platform)

{

return sprintf('BINARY(%d)', $fieldDeclaration['length']);

}

public function getName()

{

return self::BINARY;

}

public function convertToPhpValue($value, AbstractPlatform $platform)

{

if ($value !== null) {

$value= unpack('H*', $value);

$hash = array_shift($value);

$uuid = substr($hash, 0, 8) . '-' . substr($hash, 8, 4) . '-' . substr($hash, 12, 4) . '-' . substr($hash, 16, 4) . '-' . substr($hash, 20, 12);

return $uuid;

}

public function convertToDatabaseValue($value, AbstractPlatform $platform)

{

if ($value !== null) {

return pack('H*', str_replace('-', '',$value));

}

It turned out that doing a doctrine:schema:update kept executing the same update query over and over again:

$ app/console doctrine:schema:update --dump-sql
ALTER TABLE Group CHANGE uuId uuId BINARY(16) DEFAULT NULL;

1 2	$ app/console doctrine:schema:update --dump-sql ALTER TABLE Group CHANGE uuId uuId BINARY(16) DEFAULT NULL;

One hour of debugging later I discovered the issue originated to MySqlSchemaManager::_getPortableTableColumnDefinition where a MySQL column gets reverse-engineered into a Doctrine\DBAL\Types\Type which is used in the schema comparison tool. We’re storing the UUID in a BINARY(16) field which results in a BinaryType after reverse-engineering because Doctrine can’t tell the difference between a BinaryType and UuidType because both result in the exact same MySQL column definition. However, there is a solution to fix this.

Using DC2Type in the comment

The solution is to add a comment to the field to store the metadata in. This seems to be missing in the docs but I’ve found some JIRA issue describing the feature. We have to change our column definition so the metadata of the type doesn’t get lost:

ALTER TABLE Group CHANGE uuId uuId BINARY(16) COMMENT '(DC2Type:uuid)' DEFAULT NULL;

1	ALTER TABLE Group CHANGE uuId uuId BINARY(16) COMMENT '(DC2Type:uuid)' DEFAULT NULL;

The corrected UuidType looks as following:

namespace Wb\Component\Doctrine\DBAL\Types;

use Doctrine\DBAL\Types\BinaryType;
use Doctrine\DBAL\Types\Type;
use Doctrine\DBAL\Platforms\AbstractPlatform;

/**
 * Type that maps a PHP array to a clob SQL type.
 *
 * @since 2.0
 * @link https://gist.github.com/Sitebase/5013494
 */
class UuidType extends BinaryType
{

    /**
     * @param array            $fieldDeclaration
     * @param AbstractPlatform $platform
     * @return string
     */
    public function getSqlDeclaration(array $fieldDeclaration, AbstractPlatform $platform)
    {
        return sprintf('BINARY(%d) COMMENT \'(DC2Type:uuid)\'', $fieldDeclaration['length']);
    }

    /**
     * @return string
     */
    public function getName()
    {
        return 'uuid';
    }

    /**
     * @param mixed            $value
     * @param AbstractPlatform $platform
     * @return string
     */
    public function convertToPhpValue($value, AbstractPlatform $platform)
    {
        if ($value !== null) {
            $value= unpack('H*', $value);
            $hash = array_shift($value);
            $uuid = substr($hash, 0, 8) . '-' . substr($hash, 8, 4) . '-' . substr($hash, 12, 4) . '-' .
                substr($hash, 16, 4) . '-' . substr($hash, 20, 12);

            return $uuid;
        }
    }

    /**
     * @param mixed            $value
     * @param AbstractPlatform $platform
     * @return string
     */
    public function convertToDatabaseValue($value, AbstractPlatform $platform)
    {
        if ($value !== null) {
            return pack('H*', str_replace('-', '', $value));
        }
    }
}

namespace Wb\Component\Doctrine\DBAL\Types;

use Doctrine\DBAL\Types\BinaryType;

use Doctrine\DBAL\Types\Type;

use Doctrine\DBAL\Platforms\AbstractPlatform;

/**

* Type that maps a PHP array to a clob SQL type.

* @since 2.0

* @link https://gist.github.com/Sitebase/5013494

class UuidType extends BinaryType

{

/**

* @param array $fieldDeclaration

* @param AbstractPlatform $platform

* @return string

public function getSqlDeclaration(array $fieldDeclaration, AbstractPlatform $platform)

{

return sprintf('BINARY(%d) COMMENT \'(DC2Type:uuid)\'', $fieldDeclaration['length']);

}

/**

* @return string

public function getName()

{

return 'uuid';

}

/**

* @param mixed $value

* @param AbstractPlatform $platform

* @return string

public function convertToPhpValue($value, AbstractPlatform $platform)

{

if ($value !== null) {

$value= unpack('H*', $value);

$hash = array_shift($value);

$uuid = substr($hash, 0, 8) . '-' . substr($hash, 8, 4) . '-' . substr($hash, 12, 4) . '-' .

substr($hash, 16, 4) . '-' . substr($hash, 20, 12);

return $uuid;

}

/**

* @param mixed $value

* @param AbstractPlatform $platform

* @return string

public function convertToDatabaseValue($value, AbstractPlatform $platform)

{

if ($value !== null) {

return pack('H*', str_replace('-', '', $value));

}

Unfortunately the binary type is always reverse-engineered with $fixed = true so you have to configure the UuidType accordingly (note the options) on your entities (haven’t found a better way yet):

<?php

class Group 
{
    /**
     * @var string $uuId
     *
     * @ORM\Column(type="uuid", length=16, options={"fixed"=true}, nullable=true)
     */
    private $uuId;
}

<?php

class Group

{

/**

* @var string $uuId

* @ORM\Column(type="uuid", length=16, options={"fixed"=true}, nullable=true)

private $uuId;

}

This comment stuff in Doctrine is not well-documented and probably a lot of people experienced the same behaviour in the schema tool so I hope I prevented some nasty debug sessions in Doctrine’s core (although you learn a lot from it).

Posted on June 25, 2015June 25, 2015 by ricbra 4

CQRS: How to handle file uploads?

If you are like me one who tries to keep up with all the cool stuff happening in the PHP world you’ve probably noticed the buzz around Domain Driven Design and more recently Event Sourcing and CQRS. Last year Qandidate released Broadway: a project providing infrastructure and helpers for introducing CQRS and Event Sourcing into your PHP stack. It wasn’t until last month before I got the change to get my hands on it. We adopted the framework in one of the latest projects at work. And it didn’t take long before we ran into all kind of problems and questions 🙂 .

So for every question we have I’ll try to write a blogpost so others can learn. Also I’m curious about how you handle the problems I describe in these posts, so don’t hesitate to comment if you have a different opinion. Let’s dive into what should be the first in a series of post about CQRS!

The problem

In the application we’re building one requirement is that users can configure attachments to be send to a user when performing some kind of action. We’re using Symfony2 and Broadway so I our code will be very specific to these frameworks. Consider the following form:

namespace Wb\PoolBundle\Form\Type;

use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\FormBuilderInterface;
use Symfony\Component\Validator\Constraints\File;
use Symfony\Component\Validator\Constraints\NotBlank;

class UploadAttachmentFormType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder
            ->add('title', 'text', [
                'label'         => 'Naam',
                'required'      => true,
                'constraints'   => [
                    new NotBlank()
                ]
            ])
            ->add('file', 'file', [
                'required'      => true,
                'label'         => 'Bestand',
                'constraints'   => [
                    new NotBlank(),
                    new File([
                        'maxSize'   => '4M',
                        'mimeTypes' => [
                            'application/pdf',
                            'application/x-pdf',
                            'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
                            'application/x-msword',
                            'application/msword',
                            'application/zip',
                            'image/png',
                            'image/jpeg',
                            'image/pjpeg',
                            'image/gif'
                        ]
                    ])
                ]
            ]);
    }
    
    public function getName()
    {
        return 'upload_attachment';
    }
}

namespace Wb\PoolBundle\Form\Type;

use Symfony\Component\Form\AbstractType;

use Symfony\Component\Form\FormBuilderInterface;

use Symfony\Component\Validator\Constraints\File;

use Symfony\Component\Validator\Constraints\NotBlank;

class UploadAttachmentFormType extends AbstractType

{

public function buildForm(FormBuilderInterface $builder, array $options)

{

$builder

->add('title', 'text', [

'label' => 'Naam',

'required' => true,

'constraints' => [

new NotBlank()

]

])

->add('file', 'file', [

'required' => true,

'label' => 'Bestand',

'constraints' => [

new NotBlank(),

new File([

'maxSize' => '4M',

'mimeTypes' => [

'application/pdf',

'application/x-pdf',

'application/vnd.openxmlformats-officedocument.wordprocessingml.document',

'application/x-msword',

'application/msword',

'application/zip',

'image/png',

'image/jpeg',

'image/pjpeg',

'image/gif'

]

])

]

]);

}

public function getName()

{

return 'upload_attachment';

}

In the controller we validate the form, construct our UploadAttachment command – which is just a DTO – by passing all the values from to form to the command bus:

class AttachmentController
{    
    public function uploadAction(Request $request, $poolId)
    {
        $form = $this->formFactory->create(new UploadAttachmentFormType());

        if ($form->handleRequest($request)->isValid()) {
            $data    = $form->getData();
            $command = new UploadAttachment(
                new PoolId($poolId),
                new AttachmentId($this->uuidGenerator->generate()),
                $data['file']
                $data['title']
            );

            $this->commandBus->dispatch($command);

            $request->getSession()->getFlashBag()->add('notice', sprintf(
                'Attachment "%s" has been uploaded.',
                $title
            ));

            return new RedirectResponse($this->router->generate('pool_configure_procedure', [
                'poolId' => $poolId
            ]));
        }

        return new Response($this->twig->render('AcmeBundle:Form:default.html.twig', [
            'form'      => $form->createView(),
            'title'     => 'Upload attachment',
        ]));
    }
}

class AttachmentController

{

public function uploadAction(Request $request, $poolId)

{

$form = $this->formFactory->create(new UploadAttachmentFormType());

if ($form->handleRequest($request)->isValid()) {

$data = $form->getData();

$command = new UploadAttachment(

new PoolId($poolId),

new AttachmentId($this->uuidGenerator->generate()),

$data['file']

$data['title']

);

$this->commandBus->dispatch($command);

$request->getSession()->getFlashBag()->add('notice', sprintf(

'Attachment "%s" has been uploaded.',

$title

));

return new RedirectResponse($this->router->generate('pool_configure_procedure', [

'poolId' => $poolId

]));

}

return new Response($this->twig->render('AcmeBundle:Form:default.html.twig', [

'form' => $form->createView(),

'title' => 'Upload attachment',

]));

}

And the command handler calls the appropriate method on our aggregate:

namespace Wb\Pool\Model\Procedure;

use Broadway\CommandHandling\CommandHandler;
use Wb\Pool\Model\Pool\PoolRepository;
use Wb\Pool\Model\Pool\Pool;

class AttachmentCommandHandler extends CommandHandler
{
    /**
     * @var PoolRepository
     */
    private $poolRepository;

    public function __construct(PoolRepository $poolRepository)
    {
        $this->poolRepository = $poolRepository;
    }

    public function handleUploadAttachment(UploadAttachment $command)
    {
        $pool = $this->poolRepository->load($command->getPoolId());

        $procedure = $pool->getProcedure();
        $procedure->uploadAttachment(
            $command->getAttachmentId(),
            $command->getFile(),
            $command->getTitle(),
            $command->getPoolId()
        );

        $this->poolRepository->save($pool);
    }
}

namespace Wb\Pool\Model\Procedure;

use Broadway\CommandHandling\CommandHandler;

use Wb\Pool\Model\Pool\PoolRepository;

use Wb\Pool\Model\Pool\Pool;

class AttachmentCommandHandler extends CommandHandler

{

/**

* @var PoolRepository

private $poolRepository;

public function __construct(PoolRepository $poolRepository)

{

$this->poolRepository = $poolRepository;

}

public function handleUploadAttachment(UploadAttachment $command)

{

$pool = $this->poolRepository->load($command->getPoolId());

$procedure = $pool->getProcedure();

$procedure->uploadAttachment(

$command->getAttachmentId(),

$command->getFile(),

$command->getTitle(),

$command->getPoolId()

);

$this->poolRepository->save($pool);

}

Our aggregate creates a new event:

class Pool
{
    public function uploadAttachment(
        AttachmentId $attachmentId,
        $file,
        $title,
        PoolId $poolId
    ) {
        $this->apply(new AttachmentUploaded(
            $attachmentId,
            $file,
            $title,
            $poolId
        ));
    }
}

class Pool

{

public function uploadAttachment(

AttachmentId $attachmentId,

$file,

$title,

PoolId $poolId

) {

$this->apply(new AttachmentUploaded(

$attachmentId,

$file,

$title,

$poolId

));

}

But as you probably noticed now we run into problems because we’re passing around a UploadedFile instance in an event. Imagine how this would get stored into the event store:

class AttachmentUploaded implements SerializableInterface
{
    // ...
    public function serialize()
    {
        return [
            'attachmentId'      => (string) $this->getAttachmentId(),
            'title'             => $this->getTitle(),
            'file'              => file_get_contents($file->getPathname()) // This isn't gonna work
            'poolId'            => (string) $this->getPoolId()
        ];
    }
    // ...
}

class AttachmentUploaded implements SerializableInterface

{

// ...

public function serialize()

{

return [

'attachmentId' => (string) $this->getAttachmentId(),

'title' => $this->getTitle(),

'file' => file_get_contents($file->getPathname()) // This isn't gonna work

'poolId' => (string) $this->getPoolId()

];

}

// ...

}

Storing the complete file in the event storage is theoretically possible but we prefer to store our files not in MySQL but somewhere in a S3 bucket in the cloud. If you do your event store will grow quickly and you’ll have other challenges to wrap your head around. Keep in mind events often will be transferred by some queue like RabbitMQ.

After some digging around on the internet I found some others with the same problem. On Freenode #qandidate I also asked for advice. In general everybody stores the file in the controller or command handler and passes on the id to the event.

The solution

We’ve chosen to store the file in our controller and pass on the UUID to the command. A code example is worth a thousand words:

class AttachmentController
{    
    public function uploadAction(Request $request, $poolId)
    {
        $form = $this->formFactory->create(new UploadAttachmentFormType());

        if ($form->handleRequest($request)->isValid()) {
            $data    = $form->getData();
            $file    = $data['file'];
            $title   = $data['title'];
            // Construct our command with some data we need from the file
            $command = new UploadAttachment(
                new PoolId($poolId),
                new AttachmentId($this->uuidGenerator->generate()),
                $file->getClientOriginalName(), // To display in backend etc.
                $this->fileStorage->put($file), // Store file in S3, return filename
                $file->getClientMimeType(), 
                $title
            );

            $this->commandBus->dispatch($command);

            $request->getSession()->getFlashBag()->add('notice', sprintf(
                'Attachment "%s" uploaded.',
                $title
            ));

            return new RedirectResponse($this->router->generate('pool_configure_procedure', [
                'poolId' => $poolId
            ]));
        }

        return new Response($this->twig->render('AcmeBundle:Form:default.html.twig', [
            'form'      => $form->createView(),
            'title'     => 'Upload attachment',
        ]));
    }
}

class AttachmentController

{

public function uploadAction(Request $request, $poolId)

{

$form = $this->formFactory->create(new UploadAttachmentFormType());

if ($form->handleRequest($request)->isValid()) {

$data = $form->getData();

$file = $data['file'];

$title = $data['title'];

// Construct our command with some data we need from the file

$command = new UploadAttachment(

new PoolId($poolId),

new AttachmentId($this->uuidGenerator->generate()),

$file->getClientOriginalName(), // To display in backend etc.

$this->fileStorage->put($file), // Store file in S3, return filename

$file->getClientMimeType(),

$title

);

$this->commandBus->dispatch($command);

$request->getSession()->getFlashBag()->add('notice', sprintf(

'Attachment "%s" uploaded.',

$title

));

return new RedirectResponse($this->router->generate('pool_configure_procedure', [

'poolId' => $poolId

]));

}

return new Response($this->twig->render('AcmeBundle:Form:default.html.twig', [

'form' => $form->createView(),

'title' => 'Upload attachment',

]));

}

Drawbacks

There are a couple of drawbacks in this method:

every new attachment results in a new file, this could take up a lot of storage from unused files
if something goes wrong in the command handler, the file is stored already

Personally I see it as a benefit we have a history of every single attachment uploaded. We can easily go back in time and revert an erroneous upload or debug what our users did wrong in case of a problem.

By only passing around the UUID our event keeps small and this makes it easy to be published on RabbitMQ.

Posted on June 9, 2015June 10, 2015 by ricbra 2

Debugging Selenium with X11 Forwarding on Scrutinizer CI

Last week we ran into some issues on Scrutinizer CI with our Behat Selenium test suite. These things tend to be quite hard to debug: as Selenium is running headless in X virtual framebuffer (Xvfb) there is nothing to see for the developer. It’s possible to take screenshots, but this requires code changes (probably).

X11 Forwarding

One of the cool things you can do with Selenium (or more particular Firefox or Chrome) is X11 forwarding. If you’re running a X.Org Window system it is possible to forward the display from one box to another. When you’re on a Linux desktop environment you’re golden, but if you’re on a Mac like me you have to install XQuartz to get it working. Follow the instructions on the site and don’t forget to logout and login after installation otherwise your $DISPLAY enviroment variable is empty. By the way: if you’re a Windows user I honestly suggest to get a Mac or install Ubuntu to get it working :).

If you want more information on this topic there is plenty of information to be found on the interwebz.

SSH Remote debugging session

To get this X11 forwarding working on Scrutinizer CI first request a new SSH debugging session. This can be done on the inspection page. When the inspection fails you can retry but in the same dropdown also choose for “SSH Remote debugging”. The first time you do this it will ask for your public keys from Github, you should accept the request. It may take some time but after a few seconds or minutes you’ll receive a SSH login to connect to. Add an -X switch after ssh, so the command looks like this:

$ ssh -X user@11.22.33.44 -p 3312

1	$ ssh -X user@11.22.33.44 -p 3312

Now login on the remote machine and verify your $DISPLAY server is set (should be something like localhost:10.0). Open ~/.profile in your favourite editor and remove the line:

export SCREEN=":99"

1	export SCREEN=":99"

Do not log out, but create new SSH session with the command above. If all is fine you should be able to start firefox and see the browser appearing. Kill this firefox instance, and start Selenium in this session (java -jar /location/of/selenium.jar) and in the first session start your Behat tests.

Posted on April 15, 2015April 30, 2015 by ricbra

Speedup your test suite on Codeship using ParallelCI

As I mentioned in an earlier blog post we use Codeship to test some of our private repositories. The folks at Codeship improved their service a lot since we first used it: the UI is improved a lot (both visually as practically) and the list of notification services keeps growing too.

Lately they introduced a cool new feature called ParallelCI. Travis CI has a similar feature called build matrix. You can split up your test suite in multiple parallel builds, called pipelines. If you have a big or slow test suite (probably your Behat tests) you can speed up things a lot by splitting them into multiple pipelines.

Example configuration

Because our phpspec suite is fast enough, we’ve splitted our Behat suite in multiple pipelines. Of course this is project dependant and will vary per use case. To enable ParallelCI open your project settings at Codeship and click on the “Test” link. Scroll down for the “Configure Test Pipelines” section. There will be one pipeline configured called “Test commands” in which all your current test commands are configured.
Click on the green “Add new pipeline” link and a new pipeline tab will be added. Give it a clear name and add your test command. To get an idea of how this can be done take a look at our configuration:

Tab #1: Behat user

bin/behat --suite=user

1	bin/behat --suite=user

Tab #2: Behat profile

bin/behat --suite=profile

1	bin/behat --suite=profile

Tab #3: phpspec

bin/phpspec run --format=pretty --no-code-generation

1	bin/phpspec run --format=pretty --no-code-generation

When you save these settings (the pipeline edit form is bit cumbersome as you will notice when adding new tabs, but I guess this will be improved soon enough) and rerun your last run you’ll see your suite will be split into multiple pipelines and as a result it will speedup things drastically. So I definitely see the use of this new feature and I’m sure you’ll love it for your bigger test suites.

Posted on February 12, 2015February 12, 2015 by ricbra

Symfony2 and RabbitMQ: Lessons learned

Last year we introduced RabbitMQ into our stack at Waarneembemiddeling.nl. We were in desperate need of a worker queue and after fiddling around with Gearman, Beanstalkd and RabbitMQ we made our choice: RabbitMQ it will be.

Now there’s quite some information to be found on RabbitMQ and how to use it, but a lot of things you have to find out yourself. Questions like:

what happens to messages on the queue after a service restart?
what happens to messages on the queue after a reboot?
how do we notice that a worker crashed?
what happens to my message when the consumer dies while processing?
etc.

Using RabbitMQ and Symfony2 (or php in general) is quite easy. There is a bundle for Symfony2 called OldSoundRabbitMqBundle and a php library called php-amqplib which work very well. Both are from the same author, you should probably thank him for that 🙂 .

First try: pure php consumers

We’re running a fairly common setup. Because we’ve been warned that php consumer die out every now and then, we’re using Supervisor to start new consumers when needed. There is a lot of information out there on this subject so I won’t go in there.

Despite the warnings we started with pure php consumers powered by the commands in OldSoundRabbitMqBundle. The first workers were started like this:

$ app/console rabbitmq:consumer async_event -e=prod

1	$ app/console rabbitmq:consumer async_event -e=prod

This means we’re consuming from the async_event queue without any limit to the messages. Basically this means it will run forever, or better said: until php crashes. Or worse: your consumer ends up in non-response state. Which means it doesn’t process any message any more and Supervisor thinks all is fine because you still have a running process. This happened once to our mail queue. I can assure you it’s better to prevent these kind of things.

Second try: pure php consumers with limited messages

So after the mail-gate I was searching for a quick way to make our setup more error proof. The OldSoundRabbitMqBundle supports limiting the messages to process. So I limited our workers so that they got restarted a couple of times a day:

$ app/console rabbitmq:consumer async_event -e=prod -m 10

1	$ app/console rabbitmq:consumer async_event -e=prod -m 10

After that things got running more smoothly and it took a while before we encountered new problems. While spitting trough the logs I notices some consumers produced some errors. A brief summary:

General error: 2006 MySQL server has gone away
Warning: Error while sending QUERY packet.

Because the consumer is one process that keeps running, that also means that the service container and stuff keeps existing in memory. When you’ve done some queries the database connection keeps open in the background. And if it’s quiet on our queue, it may take some time before we reach the message limit. If that time exceeds the connect_timeout of your MySQL server, you’ll run into the warnings and errors about lost connections.

Of course we should close the connection after the message is processed or could try catch for Doctrine DBAL connection exceptions or increase the connect_timeout setting but thats just denying the real problem. Running consumers with a booted Symfony2 kernel just doesn’t work so well.

A final resort could be to strip down the consumers and don’t use the Symfony2 kernel and container but we don’t like that. Our messages are most of the time serialized events which get dispatched again after the consumer picks them up. At the application level we don’t want to know wether we are in a RabbitMQ consumer or in a normal HTTP request.

Real solution: rabbitmq-cli-consumer

So it took a couple of months to learn the hard way we needed some different solution for our consumers. I found this interesting blog post about the same problem. He solved it with Java and Ruby consumers. We all learned java in college right, but I don’t like to run the memory eating jvm on our servers. The Ruby consumer unfortunately misses some good documenten for me as Ruby virgin. So I got a bit lost there.

That was the point where Go got in. Go is a kind of improved C with not real OO but a lot of cool stuff in it. I wrote a application that makes it possible to consume messages from RabbitMQ queue and pipe them into an command line application. I called it: rabbitmq-cli-consumer.

The main advantages for using rabbitmq-cli-consumer are:

no more stability issues to deal with
lightweight and fast
no need to restart your workers after a fresh deployment

We still use supervisor to start and stop the consumers because it’s the right tool for it. An example of how we start a consumer:

$ rabbitmq-cli-consumer -c example.conf -e "/home/vagrant/current/app/console wb:consumer:mail --env=prod" -V
2015/01/09 20:57:52 Setting QoS... 
2015/01/09 20:57:52 Succeeded setting QoS.
2015/01/09 20:57:52 Registering consumer... 
2015/01/09 20:57:52 Succeeded registering consumer.
2015/01/09 20:57:52 Waiting for messages...
2015/01/09 20:58:07 Processing message...
2015/01/09 20:58:13 Processed!
...

$ rabbitmq-cli-consumer -c example.conf -e "/home/vagrant/current/app/console wb:consumer:mail --env=prod" -V

2015/01/09 20:57:52 Setting QoS...

2015/01/09 20:57:52 Succeeded setting QoS.

2015/01/09 20:57:52 Registering consumer...

2015/01/09 20:57:52 Succeeded registering consumer.

2015/01/09 20:57:52 Waiting for messages...

2015/01/09 20:58:07 Processing message...

2015/01/09 20:58:13 Processed!

...

An example of a Symfony2 command we use:

namespace Wb\Bundle\CoreBundle\Command;

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class MailConsumerCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('wb:consumer:mail')
            ->addArgument('message', InputArgument::REQUIRED)
            ;
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $message = $input->getArgument('message');
        if (! $message = unserialize(base64_decode($message))) {
            throw new \InvalidArgumentException('Invalid input received');
        }

        $container = $this->getContainer();
        $mailer = $container->get('swiftmailer.mailer.mandrill');
        $transport = $container->get('swiftmailer.mailer.mandrill.transport.real');
        $logger = $container->get('rabbitmq_mail_logger');

        $logger->info(sprintf(
            '[%s] [%s] Received message from queue',
            $message->getSubject(), implode(', ', array_keys($message->getTo()))
        ));
        $spool = $mailer->getTransport()->getSpool();
        $mailer->send($message);
        $spool->flushQueue($transport);

        $logger->info(sprintf(
            '[%s] [%s] Mail send',
            $message->getSubject(), implode(', ', array_keys($message->getTo()))
        ));
    }
}

namespace Wb\Bundle\CoreBundle\Command;

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;

use Symfony\Component\Console\Input\InputArgument;

use Symfony\Component\Console\Input\InputInterface;

use Symfony\Component\Console\Output\OutputInterface;

class MailConsumerCommand extends ContainerAwareCommand

{

protected function configure()

{

$this

->setName('wb:consumer:mail')

->addArgument('message', InputArgument::REQUIRED)

;

}

protected function execute(InputInterface $input, OutputInterface $output)

{

$message = $input->getArgument('message');

if (! $message = unserialize(base64_decode($message))) {

throw new \InvalidArgumentException('Invalid input received');

}

$container = $this->getContainer();

$mailer = $container->get('swiftmailer.mailer.mandrill');

$transport = $container->get('swiftmailer.mailer.mandrill.transport.real');

$logger = $container->get('rabbitmq_mail_logger');

$logger->info(sprintf(

'[%s] [%s] Received message from queue',

$message->getSubject(), implode(', ', array_keys($message->getTo()))

));

$spool = $mailer->getTransport()->getSpool();

$mailer->send($message);

$spool->flushQueue($transport);

$logger->info(sprintf(

'[%s] [%s] Mail send',

$message->getSubject(), implode(', ', array_keys($message->getTo()))

));

}

Final tip: use the management plugin

Before even starting with RabbitMQ make sure you have the management plugin installed. It gives you a good overview about whats happening. Also you can purge queues, add users, add vhosts etc.

Posted on January 9, 2015May 7, 2015 by ricbra 13

Install Selenium headless on Debian Wheezy (optionally with Ansible)

When you start testing with Behat and Mink Selenium2 driver you also need a browser running. Because we develop on a virtualised server installing FireFox was a bit more tricky then I expected. Of course a search yielded some interesting results, but also a lot of crap. So here is a little writeup on how I managed to get it running to save you some time. An example playbook can be found at the bottom of this post. But beware: this is Debian only!

On Debian there is a package called iceweasel. This is a rebranded version of FireFox. Because of that there is no firefox package available in the default repositories.

We are using Ansible for configuration management (for both our production and develop environment) so I prefer a package above compiling shit because that’s much easier to automate. There are a couple of options to install FireFox trough package manager:

add Linux Mint repository
add ubuntuzilla repository

Using the Linux Mint repository I experienced some problems. The Ubuntuzilla repository worked like a charm. If you want to install manually just follow the instructions in their Wiki. After adding the repository you can install the firefox package:

$ sudo apt-get install firefox-mozilla-build

1	$ sudo apt-get install firefox-mozilla-build

To run Firefox headless you also need some display server and to emulate that we are going to use xvfb. Selenium requires Java, thus we install:

$ sudo apt-get install xvfb openjdk-7-jre-headless

1	$ sudo apt-get install xvfb openjdk-7-jre-headless

Download Selenium somewhere:

$ mkdir /opt/selenium
$ wget -O /opt/selenium/selenium.jar http://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar

1 2	$ mkdir /opt/selenium $ wget -O /opt/selenium/selenium.jar http://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar

You should be able to start Selenium now:

$ xvfb-run java -jar /opt/selenium/selenium.jar

1	$ xvfb-run java -jar /opt/selenium/selenium.jar

Starting by hand is a bit lame, so we use this startup script:

### BEGIN INIT INFO
# Provides:          selenium
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: selenium
# Description:       selenium with xvfb
### END INIT INFO

XVFB=/usr/bin/xvfb-run
XVFBARGS="java -jar /opt/selenium/selenium.jar"
PIDFILE=/var/run/xvfb.pid
case "$1" in
  start)
    echo -n "Starting virtual X frame buffer: Xvfb"
    start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $XVFB -- $XVFBARGS
    echo "."
    ;;
  stop)
    echo -n "Stopping virtual X frame buffer: Xvfb"
    # does not work with start-stop-daemon because of child processes
    read pid <$PIDFILE
    pkill -TERM -P $pid
    echo "."
    ;;
  restart)
    $0 stop
    $0 start
    ;;
  *)
        echo "Usage: /etc/init.d/selenium {start|stop|restart}"
        exit 1
esac

exit 0

### BEGIN INIT INFO

# Provides: selenium

# Required-Start: $local_fs $network

# Required-Stop: $local_fs

# Default-Start: 2 3 4 5

# Default-Stop: 0 1 6

# Short-Description: selenium

# Description: selenium with xvfb

### END INIT INFO

XVFB=/usr/bin/xvfb-run

XVFBARGS="java -jar /opt/selenium/selenium.jar"

PIDFILE=/var/run/xvfb.pid

case "$1" in

start)

echo -n "Starting virtual X frame buffer: Xvfb"

start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $XVFB -- $XVFBARGS

echo "."

;;

stop)

echo -n "Stopping virtual X frame buffer: Xvfb"

# does not work with start-stop-daemon because of child processes

read pid <$PIDFILE

pkill -TERM -P $pid

echo "."

;;

restart)

$0 stop

$0 start

;;

echo "Usage: /etc/init.d/selenium {start|stop|restart}"

exit 1

esac

exit 0

Copy this to /etc/init.d/selenium and after that you can:

$ sudo service selenium start

1	$ sudo service selenium start

And when we create an Ansible playbook for all this we get:

---
- name: Instal Selenium
  tasks:
  - apt_repository: repo="deb http://downloads.sourceforge.net/project/ubuntuzilla/mozilla/apt all main" state=present
  - apt_key: id="C1289A29" keyserver=keyserver.ubuntu.com state=present
  - apt: name=iceweasel state=absent
  - apt: name={{ item }} state=present update_cache=yes
    with_items:
    - firefox-mozilla-build
    - openjdk-7-jre-headless
    - xvfb
  - file: name=/opt/selenium state=directory mode=0755
  - get_url: url="http://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar" dest=/opt/selenium/selenium.jar
  - copy: src=selenium dest=/etc/init.d/selenium mode=0755

---

- name: Instal Selenium

tasks:

- apt_repository: repo="deb http://downloads.sourceforge.net/project/ubuntuzilla/mozilla/apt all main" state=present

- apt_key: id="C1289A29" keyserver=keyserver.ubuntu.com state=present

- apt: name=iceweasel state=absent

- apt: name={{ item }} state=present update_cache=yes

with_items:

- firefox-mozilla-build

- openjdk-7-jre-headless

- xvfb

- file: name=/opt/selenium state=directory mode=0755

- get_url: url="http://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar" dest=/opt/selenium/selenium.jar

- copy: src=selenium dest=/etc/init.d/selenium mode=0755

Posted on November 5, 2014April 28, 2015 by ricbra 5

How to use Codeship with Symfony2, phpspec and Behat

My coworkers and I at waarneembemiddeling.nl are really fond of phpspec and Behat. Yes, we must confess: we didn’t test much since a couple of months ago. We skipped the phpunit age and started right away with phpspec and Behat. We also like services, so instead of setting up (and maintain) our own CI server, we use Codeship. To be honest we fell in love with Travis, but that was a little bit to expensive for us. And so our search ended at Codeship.

There is some documentation on how to use it with php, but its not that in depth about phpspec and friends. Let’s start with phpspec, as this is pretty easy. I’m assuming you install phpspec and Behat as dev dependencies using Composer:

{
    "require": {
    	
    },
    "require-dev": {
        "sensio/generator-bundle":           "~2.3",
        "phpspec/phpspec":                   "~2.0",
        "behat/behat":                       "~3.0",
        "behat/symfony2-extension":          "~2.0@dev",
        "behat/mink-extension":              "~2.0@dev",
        "behat/mink-browserkit-driver":      "~1.2@dev",
        "behat/mink-selenium2-driver":       "@dev",
        "behat/mink":                        "~1.6@dev"
    }
}

{

"require": {

"require-dev": {

"sensio/generator-bundle": "~2.3",

"phpspec/phpspec": "~2.0",

"behat/behat": "~3.0",

"behat/symfony2-extension": "~2.0@dev",

"behat/mink-extension": "~2.0@dev",

"behat/mink-browserkit-driver": "~1.2@dev",

"behat/mink-selenium2-driver": "@dev",

"behat/mink": "~1.6@dev"

}

phpspec

Now head over to codeship.com and edit your projects configuration. Pick “PHP” as your technology (didn’t see that one coming). In the “setup commands” field we first select the desired php version:

phpenv local 5.5

1	phpenv local 5.5

Next install deps (I believe this line is placed there by default by the codeship guys):

composer install --prefer-source --no-interaction

1	composer install --prefer-source --no-interaction

Then add phpsec to the “test commands” field:

bin/phpspec run --format=pretty --no-code-generation

1	bin/phpspec run --format=pretty --no-code-generation

Et voila, phpspec should now be functioning. 🙂

Behat

Behat is a little bit more difficult. The first problem you need to solve is to get the MySQL credentials into your Symfony2 application. These are provided trough environment vars, but differ from the naming convention in Symfony2.

We start by changing our app/config/config_test.yml:

imports:
    - { resource: config_dev.yml }
 
doctrine:
    dbal:
        dbname:   "%test_database_name%"
        driver:   pdo_mysql
        user:     "%test_database_user%"
        password: "%test_database_password%"
 
framework:
    test: ~
    session:
        storage_id: session.storage.mock_file
    profiler:
        collect: true
        only_exceptions: false
 
web_profiler:
    toolbar: false
    intercept_redirects: false
 
swiftmailer:
    disable_delivery: true

imports:

- { resource: config_dev.yml }

doctrine:

dbal:

dbname: "%test_database_name%"

driver: pdo_mysql

user: "%test_database_user%"

password: "%test_database_password%"

framework:

test: ~

session:

storage_id: session.storage.mock_file

profiler:

collect: true

only_exceptions: false

web_profiler:

toolbar: false

intercept_redirects: false

swiftmailer:

disable_delivery: true

Now to let Symfony2 pick up the environment vars we have to follow the convention I just mentioned. This means that an environment variable with the name SYMFONY__TEST_DATABASE_USER will be recognised when building the container. But let’s start by adding a bash script to ease the setup of the testing environment (locally and Codeship). Call it setup_test_env.sh and place it in the root of your project:

#!/usr/bin/env sh
 
# we need to define our credentials here via environment vars as thats the way we get mysql
# credentials in Codeship.
if [ -z $SYMFONY__TEST_DATABASE_USER ]; then
    export SYMFONY__TEST_DATABASE_USER=root
fi
if [ -z $SYMFONY__TEST_DATABASE_PASSWORD ]; then
    export SYMFONY__TEST_DATABASE_PASSWORD=yourdevweakdbpass
fi
if [ -z $SYMFONY__TEST_DATABASE_NAME ]; then
    export SYMFONY__TEST_DATABASE_NAME=test
fi
 
# Setup DB and clear cache
php ./app/console --env=test doctrine:database:drop --force
php ./app/console --env=test doctrine:database:create
php ./app/console --env=test doctrine:migrations:migrate --no-interaction
php ./app/console --env=test cache:clear

#!/usr/bin/env sh

# we need to define our credentials here via environment vars as thats the way we get mysql

# credentials in Codeship.

if [ -z $SYMFONY__TEST_DATABASE_USER ]; then

export SYMFONY__TEST_DATABASE_USER=root

if [ -z $SYMFONY__TEST_DATABASE_PASSWORD ]; then

export SYMFONY__TEST_DATABASE_PASSWORD=yourdevweakdbpass

if [ -z $SYMFONY__TEST_DATABASE_NAME ]; then

export SYMFONY__TEST_DATABASE_NAME=test

# Setup DB and clear cache

php ./app/console --env=test doctrine:database:drop --force

php ./app/console --env=test doctrine:database:create

php ./app/console --env=test doctrine:migrations:migrate --no-interaction

php ./app/console --env=test cache:clear

Then adjust your codeship setup commands and add:

export SYMFONY__TEST_DATABASE_USER=$MYSQL_USER
export SYMFONY__TEST_DATABASE_PASSWORD=$MYSQL_PASSWORD
export SYMFONY__TEST_DATABASE_NAME=test
./setup_test_env.sh

export SYMFONY__TEST_DATABASE_USER=$MYSQL_USER

export SYMFONY__TEST_DATABASE_PASSWORD=$MYSQL_PASSWORD

export SYMFONY__TEST_DATABASE_NAME=test

./setup_test_env.sh

Last but not least add the behat command to the test commands:

bin/behat -n

1	bin/behat -n

Things should be working now. Quickly enough you will run into the infamous xdebug “Fatal error: Maximum function nesting level of ‘100’ reached” error. Let’s fix this right away and add this in your setup commands:

echo "xdebug.max_nesting_level=200" >> /home/rof/.phpenv/versions/5.5/etc/php.ini

1	echo "xdebug.max_nesting_level=200" >> /home/rof/.phpenv/versions/5.5/etc/php.ini

Summary

So the complete setup commands dialog for phpspec and Behat together looks like this:

phpenv local 5.5
# Install dependencies through Composer
composer install --prefer-source --no-interaction
# Fix MySQL parameters
export SYMFONY__TEST_DATABASE_USER=$MYSQL_USER
export SYMFONY__TEST_DATABASE_PASSWORD=$MYSQL_PASSWORD
export SYMFONY__TEST_DATABASE_NAME=test
echo "xdebug.max_nesting_level=200" >> /home/rof/.phpenv/versions/5.5/etc/php.ini
./setup_test_env.sh

phpenv local 5.5

# Install dependencies through Composer

composer install --prefer-source --no-interaction

# Fix MySQL parameters

export SYMFONY__TEST_DATABASE_USER=$MYSQL_USER

export SYMFONY__TEST_DATABASE_PASSWORD=$MYSQL_PASSWORD

export SYMFONY__TEST_DATABASE_NAME=test

echo "xdebug.max_nesting_level=200" >> /home/rof/.phpenv/versions/5.5/etc/php.ini

./setup_test_env.sh

And the test commands like this:

bin/phpspec run --format=pretty --no-code-generation
bin/behat -n

1 2	bin/phpspec run --format=pretty --no-code-generation bin/behat -n

Everything should be working fine now! To run your tests local don’t forget to first execute the bash script (notice the extra dot, it is required):

. ./setup_test_env.sh

1	. ./setup_test_env.sh

Happy testing! 😉

Posted on October 31, 2014January 14, 2017 by ricbra 11

Slow initialization time with Symfony2 on vagrant

A few days ago we switched our complete infrastructure from hosting provider. Also we made the switch from CentOS to Debian. So we also got a new fresh development environment using Debian and Vagrant (and latest PHP and MySQL ofcourse :)).

We expected the new dev box to be fast, but the oppositie was happening: it was slow as hell. And when I mean slow as hell, it’s terribly slow (10 – 20 seconds, also for the debug toolbar). In the past we had some more problems with performance on VirtualBox and Vagrant. There are some great post out there on this subject (here and here) which we already applied to our setup. In a nutshell:

change logs and cache dir in AppKernel
use NFS share

The cause: JMSDiExtraBundle

After some profiling I discovered there were so many calls originating from JMSDiExtraBundle I tried to disable the bundle. And guess what: loading time dropped to some whopping 200ms!

The real problem was the way the bundle was configured:

jms_di_extra:
    locations:
        all_bundles: false
        bundles: [SomeBundle, AnotherBundle]
        directories: ["%kernel.root_dir%/../src/Vendor/Model"]

jms_di_extra:

locations:

all_bundles: false

bundles: [SomeBundle, AnotherBundle]

directories: ["%kernel.root_dir%/../src/Vendor/Model"]

This causes the bundle to search trough all your php files in those locations. Apparently in the old situation (php 5.3 and CentOS) this wasn’t as problematic as in the new situation (php-fpm 5.5, Debian).

Posted on October 31, 2014April 28, 2015 by ricbra

Speed up your data migration with Spork

One of the blogs I like to keep an eye on is Kris Wallsmith his personal blog. He is a Symfony2 contributor and also author of Assetic and Buzz. Last year he wrote about a new experimental project called Spork: a wrapper around pcntl_fork to abstract away the complexities with spawning child processes with php. The article was very interesting, although I didn’t had any valid use case to try the library out. That was, until today.

It happens to be we were preparing a rather large data migration for a application with approximately 17,000 users. The legacy application stored the passwords in a unsafe way – plaintext – so we had to encrypt ’em al during the migration. Our weapon of choice was bcrypt, and using the BlowfishPasswordEncoderBundle implementing was made easy. Using bcrypt did introduce a new problem: encoding all these records would take a lot of time! That’s where Spork comes in!

Setting up the Symfony2 migration Command

If possible I wanted to fork between 8 and 15 processes to gain maximum speed. We’ll run the command on a VPS with 8 virtual cores so I want to stress the machine as much as possible ;). Unfortunately the example on GitHub as well on his blog didn’t function any more so I had to dig in just a little bit. I came up with this to get the forking working:

<?php

namespace Netvlies\AcmeMigrationBundle\Command;

use Spork\Batch\Strategy\ChunkStrategy;
use Spork\EventDispatcher\EventDispatcher;
use Spork\ProcessManager;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;

class GeneratePasswordCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('generate:password')
        ;
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $container = $this->getContainer();
        $em = $container->get('doctrine.orm.entity_manager');

        $callback = function($result) use($output) {
            $result = current($result);
            $output->writeln('Greeting from '. getmypid() .' with id '.$result['id']);
        };

        // Select all the passwords that should be encoded
        $query = $em->createQuery(
            "SELECT u.id, u.password, u.salt FROM AcmeUserBundle:User u WHERE u.isPasswordEncoded != 1 OR u.isPasswordEncoded IS NULL ORDER BY u.id"
        )->setMaxResults(10);
        $iterator = $query->iterate();

        $manager = new ProcessManager(new EventDispatcher(), true);
        $strategy = new ChunkStrategy();
        $manager->process($iterator, $callback, $strategy);
    }
}

<?php

namespace Netvlies\AcmeMigrationBundle\Command;

use Spork\Batch\Strategy\ChunkStrategy;

use Spork\EventDispatcher\EventDispatcher;

use Spork\ProcessManager;

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;

use Symfony\Component\Console\Input\InputArgument;

use Symfony\Component\Console\Input\InputInterface;

use Symfony\Component\Console\Input\InputOption;

class GeneratePasswordCommand extends ContainerAwareCommand

{

protected function configure()

{

$this

->setName('generate:password')

;

}

protected function execute(InputInterface $input, OutputInterface $output)

{

$container = $this->getContainer();

$em = $container->get('doctrine.orm.entity_manager');

$callback = function($result) use($output) {

$result = current($result);

$output->writeln('Greeting from '. getmypid() .' with id '.$result['id']);

};

// Select all the passwords that should be encoded

$query = $em->createQuery(

"SELECT u.id, u.password, u.salt FROM AcmeUserBundle:User u WHERE u.isPasswordEncoded != 1 OR u.isPasswordEncoded IS NULL ORDER BY u.id"

)->setMaxResults(10);

$iterator = $query->iterate();

$manager = new ProcessManager(new EventDispatcher(), true);

$strategy = new ChunkStrategy();

$manager->process($iterator, $callback, $strategy);

}

The command generates the following output:

$ app/console wb:generate:password
Greeting from 3743 with id 1
Greeting from 3743 with id 2
Greeting from 3743 with id 3
Greeting from 3744 with id 4
Greeting from 3744 with id 5
Greeting from 3744 with id 6
Greeting from 3745 with id 7
Greeting from 3745 with id 8
Greeting from 3745 with id 9
Greeting from 3746 with id 10

$ app/console wb:generate:password

Greeting from 3743 with id 1

Greeting from 3743 with id 2

Greeting from 3743 with id 3

Greeting from 3744 with id 4

Greeting from 3744 with id 5

Greeting from 3744 with id 6

Greeting from 3745 with id 7

Greeting from 3745 with id 8

Greeting from 3745 with id 9

Greeting from 3746 with id 10

Make it a little bit more dynamic

To be really useful I’ve added some parameters so we can control the behavior a little more. As I mentioned before I wanted to control the amount forks so I added a option to control this. This value needs to be passed on to the constructor of the ChunkStrategy:

<?php
 
namespace Netvlies\AcmeMigrationBundle\Command;
 
...
 
class GeneratePasswordCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('generate:password')
            ->addOption('forks', 'f', InputOption::VALUE_REQUIRED, 'How many childs to be spawned', 4)
        ;
    }
 
    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $forks = (int) $input->getOption('forks');
 
        ...

        $manager = new ProcessManager(new EventDispatcher(), true);
        $strategy = new ChunkStrategy($forks);
        $manager->process($iterator, $callback, $strategy);
    }
}

<?php

namespace Netvlies\AcmeMigrationBundle\Command;

...

class GeneratePasswordCommand extends ContainerAwareCommand

{

protected function configure()

{

$this

->setName('generate:password')

->addOption('forks', 'f', InputOption::VALUE_REQUIRED, 'How many childs to be spawned', 4)

;

}

protected function execute(InputInterface $input, OutputInterface $output)

{

$forks = (int) $input->getOption('forks');

...

$manager = new ProcessManager(new EventDispatcher(), true);

$strategy = new ChunkStrategy($forks);

$manager->process($iterator, $callback, $strategy);

}

I also added a max parameter so we can run some tests on a small set of users, instead of the whole database. When set I pass it on to the setMaxResults method of the $query object.

Storing the results in MySQL: Beware!

In Symfony2 projects storing and reading data from the database is pretty straight forward using Doctrine2. However when you start forking your PHP process keep in mind the following:

all the forks share the same database connection;
when the first fork exits, it will also close the database connection;
database operations in running forks will yield: General error: 2006 MySQL server has gone away

This is a known problem. In order to fix this problem I create and close a new connection in each fork:

<?php

use Doctrine\DBAL\DriverManager;
 
class GeneratePasswordCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('generate:password')
            ->addOption('forks', 'f', InputOption::VALUE_REQUIRED, 'How many childs to be spawned', 4)
            ->addOption('max', 'm', InputOption::VALUE_OPTIONAL, 'Maximum records to process')
        ;
    }
 
    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $forks = (int) $input->getOption('forks');
        $container = $this->getContainer();
        $em = $container->get('doctrine.orm.entity_manager');
        $encoder = $container->get('security.encoder.blowfish');
        $params = array(
            'dbname'    => $container->getParameter('database_name'),
            'user'      => $container->getParameter('database_user'),
            'password'  => $container->getParameter('database_password'),
            'host'      => $container->getParameter('database_host'),
            'driver'    => $container->getParameter('database_driver'),
        );
 
        $callback = function($result) use ($encoder, $output, $params) {
            // Create connection
            $conn = DriverManager::getConnection($params);
            $conn->connect();
            $result = current($result);
            $encodedPassword = $encoder->encodePassword($result['password'], $result['salt']);
            // Store result
            $stmt = $conn->prepare("UPDATE User u SET u.password = ?, u.isPasswordEncoded = 1 WHERE u.id = ?");
            $stmt->bindValue(1, $encodedPassword);
            $stmt->bindValue(2, $result['id']);
            $stmt->execute();
            // Close connection
            $conn->close();
            $output->writeln(sprintf('Encoded password for id: %d is %s', $result['id'], $encodedPassword));
        };
 
        $query = $em->createQuery(
            "SELECT u.id, u.password, u.salt FROM AcmeUserBundle:User u WHERE u.isPasswordEncoded != 1 OR u.isPasswordEncoded IS NULL ORDER BY u.id"
        );
        if ($max = $input->getOption('max')) {
            $query->setMaxResults((int) $max);
        }
        $iterator = $query->iterate();
 
        $manager = new ProcessManager(new EventDispatcher(), true);
        $strategy = new ChunkStrategy($forks);
        $manager->process($iterator, $callback, $strategy);
    }
}

<?php

use Doctrine\DBAL\DriverManager;

class GeneratePasswordCommand extends ContainerAwareCommand

{

protected function configure()

{

$this

->setName('generate:password')

->addOption('forks', 'f', InputOption::VALUE_REQUIRED, 'How many childs to be spawned', 4)

->addOption('max', 'm', InputOption::VALUE_OPTIONAL, 'Maximum records to process')

;

}

protected function execute(InputInterface $input, OutputInterface $output)

{

$forks = (int) $input->getOption('forks');

$container = $this->getContainer();

$em = $container->get('doctrine.orm.entity_manager');

$encoder = $container->get('security.encoder.blowfish');

$params = array(

'dbname' => $container->getParameter('database_name'),

'user' => $container->getParameter('database_user'),

'password' => $container->getParameter('database_password'),

'host' => $container->getParameter('database_host'),

'driver' => $container->getParameter('database_driver'),

);

$callback = function($result) use ($encoder, $output, $params) {

// Create connection

$conn = DriverManager::getConnection($params);

$conn->connect();

$result = current($result);

$encodedPassword = $encoder->encodePassword($result['password'], $result['salt']);

// Store result

$stmt = $conn->prepare("UPDATE User u SET u.password = ?, u.isPasswordEncoded = 1 WHERE u.id = ?");

$stmt->bindValue(1, $encodedPassword);

$stmt->bindValue(2, $result['id']);

$stmt->execute();

// Close connection

$conn->close();

$output->writeln(sprintf('Encoded password for id: %d is %s', $result['id'], $encodedPassword));

};

$query = $em->createQuery(

"SELECT u.id, u.password, u.salt FROM AcmeUserBundle:User u WHERE u.isPasswordEncoded != 1 OR u.isPasswordEncoded IS NULL ORDER BY u.id"

);

if ($max = $input->getOption('max')) {

$query->setMaxResults((int) $max);

}

$iterator = $query->iterate();

$manager = new ProcessManager(new EventDispatcher(), true);

$strategy = new ChunkStrategy($forks);

$manager->process($iterator, $callback, $strategy);

}

That’s basically it. Running this command on a VPS comparable with c1.xlarge Amazone EC2 server did speed up things a lot. So if you’re also working on a import job like this which can be split up in separate tasks you know what to do… Give Spork a try! It’s really easy, I promise.

UPDATE 2013-03-19
As stated in the comments by Kris, you should close the connection just before forking. Example of how to do this:

<?php
 
namespace Netvlies\AcmeMigrationBundle\Command;
 
...
use Spork\EventDispatcher\Events;
 
class GeneratePasswordCommand extends ContainerAwareCommand
{
    ...

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        ...
        // Get connection in parent
        $parentConnection = $em->getConnection();
        $dispatcher = new EventDispatcher();
        $dispatcher->addListener(Events::PRE_FORK, function() use ($parentConnection) {
            $parentConnection->close();
        });
        $manager = new ProcessManager($dispatcher, true);
        $strategy = new ChunkStrategy($forks);
        $manager->process($iterator, $callback, $strategy);
    }
}

<?php

namespace Netvlies\AcmeMigrationBundle\Command;

...

use Spork\EventDispatcher\Events;

class GeneratePasswordCommand extends ContainerAwareCommand

{

...

protected function execute(InputInterface $input, OutputInterface $output)

{

...

// Get connection in parent

$parentConnection = $em->getConnection();

$dispatcher = new EventDispatcher();

$dispatcher->addListener(Events::PRE_FORK, function() use ($parentConnection) {

$parentConnection->close();

});

$manager = new ProcessManager($dispatcher, true);

$strategy = new ChunkStrategy($forks);

$manager->process($iterator, $callback, $strategy);

}

Posted on March 17, 2013April 28, 2015 by ricbra 6

Ricbra's Blog

Loving the web since 2001

Ricbra's Blog

Category: Web development

How I use PHP generators to make my life easier

The GetAllVacanciesGenerator class

Creating a custom Doctrine DBAL type the right way

Using DC2Type in the comment

CQRS: How to handle file uploads?

The problem

The solution

Drawbacks

Debugging Selenium with X11 Forwarding on Scrutinizer CI

X11 Forwarding

SSH Remote debugging session

Speedup your test suite on Codeship using ParallelCI

Example configuration

Symfony2 and RabbitMQ: Lessons learned

First try: pure php consumers

Second try: pure php consumers with limited messages

Real solution: rabbitmq-cli-consumer

Final tip: use the management plugin

Install Selenium headless on Debian Wheezy (optionally with Ansible)

How to use Codeship with Symfony2, phpspec and Behat

phpspec

Behat

Summary

Slow initialization time with Symfony2 on vagrant

The cause: JMSDiExtraBundle

Speed up your data migration with Spork

Setting up the Symfony2 migration Command

Make it a little bit more dynamic

Storing the results in MySQL: Beware!