Speedup your test suite on Codeship using ParallelCI

As I mentioned in an earlier blog post we use Codeship to test some of our private repositories. The folks at Codeship improved their service a lot since we first used it: the UI is improved a lot (both visually as practically) and the list of notification services keeps growing too.

Lately they introduced a cool new feature called ParallelCI. Travis CI has a similar feature called build matrix. You can split up your test suite in multiple parallel builds, called pipelines. If you have a big or slow test suite (probably your Behat tests) you can speed up things a lot by splitting them into multiple pipelines.

Example configuration

Because our phpspec suite is fast enough, we’ve splitted our Behat suite in multiple pipelines. Of course this is project dependant and will vary per use case. To enable ParallelCI open your project settings at Codeship and click on the “Test” link. Scroll down for the “Configure Test Pipelines” section. There will be one pipeline configured called “Test commands” in which all your current test commands are configured.
Click on the green “Add new pipeline” link and a new pipeline tab will be added. Give it a clear name and add your test command. To get an idea of how this can be done take a look at our configuration:

Tab #1: Behat user

bin/behat --suite=user

1	bin/behat --suite=user

Tab #2: Behat profile

bin/behat --suite=profile

1	bin/behat --suite=profile

Tab #3: phpspec

bin/phpspec run --format=pretty --no-code-generation

1	bin/phpspec run --format=pretty --no-code-generation

When you save these settings (the pipeline edit form is bit cumbersome as you will notice when adding new tabs, but I guess this will be improved soon enough) and rerun your last run you’ll see your suite will be split into multiple pipelines and as a result it will speedup things drastically. So I definitely see the use of this new feature and I’m sure you’ll love it for your bigger test suites.

Posted on February 12, 2015February 12, 2015 by ricbra

Install Selenium headless on Debian Wheezy (optionally with Ansible)

When you start testing with Behat and Mink Selenium2 driver you also need a browser running. Because we develop on a virtualised server installing FireFox was a bit more tricky then I expected. Of course a search yielded some interesting results, but also a lot of crap. So here is a little writeup on how I managed to get it running to save you some time. An example playbook can be found at the bottom of this post. But beware: this is Debian only!

On Debian there is a package called iceweasel. This is a rebranded version of FireFox. Because of that there is no firefox package available in the default repositories.

We are using Ansible for configuration management (for both our production and develop environment) so I prefer a package above compiling shit because that’s much easier to automate. There are a couple of options to install FireFox trough package manager:

add Linux Mint repository
add ubuntuzilla repository

Using the Linux Mint repository I experienced some problems. The Ubuntuzilla repository worked like a charm. If you want to install manually just follow the instructions in their Wiki. After adding the repository you can install the firefox package:

$ sudo apt-get install firefox-mozilla-build

1	$ sudo apt-get install firefox-mozilla-build

To run Firefox headless you also need some display server and to emulate that we are going to use xvfb. Selenium requires Java, thus we install:

$ sudo apt-get install xvfb openjdk-7-jre-headless

1	$ sudo apt-get install xvfb openjdk-7-jre-headless

Download Selenium somewhere:

$ mkdir /opt/selenium
$ wget -O /opt/selenium/selenium.jar http://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar

1 2	$ mkdir /opt/selenium $ wget -O /opt/selenium/selenium.jar http://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar

You should be able to start Selenium now:

$ xvfb-run java -jar /opt/selenium/selenium.jar

1	$ xvfb-run java -jar /opt/selenium/selenium.jar

Starting by hand is a bit lame, so we use this startup script:

### BEGIN INIT INFO
# Provides:          selenium
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: selenium
# Description:       selenium with xvfb
### END INIT INFO

XVFB=/usr/bin/xvfb-run
XVFBARGS="java -jar /opt/selenium/selenium.jar"
PIDFILE=/var/run/xvfb.pid
case "$1" in
  start)
    echo -n "Starting virtual X frame buffer: Xvfb"
    start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $XVFB -- $XVFBARGS
    echo "."
    ;;
  stop)
    echo -n "Stopping virtual X frame buffer: Xvfb"
    # does not work with start-stop-daemon because of child processes
    read pid <$PIDFILE
    pkill -TERM -P $pid
    echo "."
    ;;
  restart)
    $0 stop
    $0 start
    ;;
  *)
        echo "Usage: /etc/init.d/selenium {start|stop|restart}"
        exit 1
esac

exit 0

### BEGIN INIT INFO

# Provides: selenium

# Required-Start: $local_fs $network

# Required-Stop: $local_fs

# Default-Start: 2 3 4 5

# Default-Stop: 0 1 6

# Short-Description: selenium

# Description: selenium with xvfb

### END INIT INFO

XVFB=/usr/bin/xvfb-run

XVFBARGS="java -jar /opt/selenium/selenium.jar"

PIDFILE=/var/run/xvfb.pid

case "$1" in

start)

echo -n "Starting virtual X frame buffer: Xvfb"

start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $XVFB -- $XVFBARGS

echo "."

;;

stop)

echo -n "Stopping virtual X frame buffer: Xvfb"

# does not work with start-stop-daemon because of child processes

read pid <$PIDFILE

pkill -TERM -P $pid

echo "."

;;

restart)

$0 stop

$0 start

;;

echo "Usage: /etc/init.d/selenium {start|stop|restart}"

exit 1

esac

exit 0

Copy this to /etc/init.d/selenium and after that you can:

$ sudo service selenium start

1	$ sudo service selenium start

And when we create an Ansible playbook for all this we get:

---
- name: Instal Selenium
  tasks:
  - apt_repository: repo="deb http://downloads.sourceforge.net/project/ubuntuzilla/mozilla/apt all main" state=present
  - apt_key: id="C1289A29" keyserver=keyserver.ubuntu.com state=present
  - apt: name=iceweasel state=absent
  - apt: name={{ item }} state=present update_cache=yes
    with_items:
    - firefox-mozilla-build
    - openjdk-7-jre-headless
    - xvfb
  - file: name=/opt/selenium state=directory mode=0755
  - get_url: url="http://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar" dest=/opt/selenium/selenium.jar
  - copy: src=selenium dest=/etc/init.d/selenium mode=0755

---

- name: Instal Selenium

tasks:

- apt_repository: repo="deb http://downloads.sourceforge.net/project/ubuntuzilla/mozilla/apt all main" state=present

- apt_key: id="C1289A29" keyserver=keyserver.ubuntu.com state=present

- apt: name=iceweasel state=absent

- apt: name={{ item }} state=present update_cache=yes

with_items:

- firefox-mozilla-build

- openjdk-7-jre-headless

- xvfb

- file: name=/opt/selenium state=directory mode=0755

- get_url: url="http://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar" dest=/opt/selenium/selenium.jar

- copy: src=selenium dest=/etc/init.d/selenium mode=0755

Posted on November 5, 2014April 28, 2015 by ricbra 5

How to use Codeship with Symfony2, phpspec and Behat

My coworkers and I at waarneembemiddeling.nl are really fond of phpspec and Behat. Yes, we must confess: we didn’t test much since a couple of months ago. We skipped the phpunit age and started right away with phpspec and Behat. We also like services, so instead of setting up (and maintain) our own CI server, we use Codeship. To be honest we fell in love with Travis, but that was a little bit to expensive for us. And so our search ended at Codeship.

There is some documentation on how to use it with php, but its not that in depth about phpspec and friends. Let’s start with phpspec, as this is pretty easy. I’m assuming you install phpspec and Behat as dev dependencies using Composer:

{
    "require": {
    	
    },
    "require-dev": {
        "sensio/generator-bundle":           "~2.3",
        "phpspec/phpspec":                   "~2.0",
        "behat/behat":                       "~3.0",
        "behat/symfony2-extension":          "~2.0@dev",
        "behat/mink-extension":              "~2.0@dev",
        "behat/mink-browserkit-driver":      "~1.2@dev",
        "behat/mink-selenium2-driver":       "@dev",
        "behat/mink":                        "~1.6@dev"
    }
}

{

"require": {

"require-dev": {

"sensio/generator-bundle": "~2.3",

"phpspec/phpspec": "~2.0",

"behat/behat": "~3.0",

"behat/symfony2-extension": "~2.0@dev",

"behat/mink-extension": "~2.0@dev",

"behat/mink-browserkit-driver": "~1.2@dev",

"behat/mink-selenium2-driver": "@dev",

"behat/mink": "~1.6@dev"

}

phpspec

Now head over to codeship.com and edit your projects configuration. Pick “PHP” as your technology (didn’t see that one coming). In the “setup commands” field we first select the desired php version:

phpenv local 5.5

1	phpenv local 5.5

Next install deps (I believe this line is placed there by default by the codeship guys):

composer install --prefer-source --no-interaction

1	composer install --prefer-source --no-interaction

Then add phpsec to the “test commands” field:

bin/phpspec run --format=pretty --no-code-generation

1	bin/phpspec run --format=pretty --no-code-generation

Et voila, phpspec should now be functioning. 🙂

Behat

Behat is a little bit more difficult. The first problem you need to solve is to get the MySQL credentials into your Symfony2 application. These are provided trough environment vars, but differ from the naming convention in Symfony2.

We start by changing our app/config/config_test.yml:

imports:
    - { resource: config_dev.yml }
 
doctrine:
    dbal:
        dbname:   "%test_database_name%"
        driver:   pdo_mysql
        user:     "%test_database_user%"
        password: "%test_database_password%"
 
framework:
    test: ~
    session:
        storage_id: session.storage.mock_file
    profiler:
        collect: true
        only_exceptions: false
 
web_profiler:
    toolbar: false
    intercept_redirects: false
 
swiftmailer:
    disable_delivery: true

imports:

- { resource: config_dev.yml }

doctrine:

dbal:

dbname: "%test_database_name%"

driver: pdo_mysql

user: "%test_database_user%"

password: "%test_database_password%"

framework:

test: ~

session:

storage_id: session.storage.mock_file

profiler:

collect: true

only_exceptions: false

web_profiler:

toolbar: false

intercept_redirects: false

swiftmailer:

disable_delivery: true

Now to let Symfony2 pick up the environment vars we have to follow the convention I just mentioned. This means that an environment variable with the name SYMFONY__TEST_DATABASE_USER will be recognised when building the container. But let’s start by adding a bash script to ease the setup of the testing environment (locally and Codeship). Call it setup_test_env.sh and place it in the root of your project:

#!/usr/bin/env sh
 
# we need to define our credentials here via environment vars as thats the way we get mysql
# credentials in Codeship.
if [ -z $SYMFONY__TEST_DATABASE_USER ]; then
    export SYMFONY__TEST_DATABASE_USER=root
fi
if [ -z $SYMFONY__TEST_DATABASE_PASSWORD ]; then
    export SYMFONY__TEST_DATABASE_PASSWORD=yourdevweakdbpass
fi
if [ -z $SYMFONY__TEST_DATABASE_NAME ]; then
    export SYMFONY__TEST_DATABASE_NAME=test
fi
 
# Setup DB and clear cache
php ./app/console --env=test doctrine:database:drop --force
php ./app/console --env=test doctrine:database:create
php ./app/console --env=test doctrine:migrations:migrate --no-interaction
php ./app/console --env=test cache:clear

#!/usr/bin/env sh

# we need to define our credentials here via environment vars as thats the way we get mysql

# credentials in Codeship.

if [ -z $SYMFONY__TEST_DATABASE_USER ]; then

export SYMFONY__TEST_DATABASE_USER=root

if [ -z $SYMFONY__TEST_DATABASE_PASSWORD ]; then

export SYMFONY__TEST_DATABASE_PASSWORD=yourdevweakdbpass

if [ -z $SYMFONY__TEST_DATABASE_NAME ]; then

export SYMFONY__TEST_DATABASE_NAME=test

# Setup DB and clear cache

php ./app/console --env=test doctrine:database:drop --force

php ./app/console --env=test doctrine:database:create

php ./app/console --env=test doctrine:migrations:migrate --no-interaction

php ./app/console --env=test cache:clear

Then adjust your codeship setup commands and add:

export SYMFONY__TEST_DATABASE_USER=$MYSQL_USER
export SYMFONY__TEST_DATABASE_PASSWORD=$MYSQL_PASSWORD
export SYMFONY__TEST_DATABASE_NAME=test
./setup_test_env.sh

export SYMFONY__TEST_DATABASE_USER=$MYSQL_USER

export SYMFONY__TEST_DATABASE_PASSWORD=$MYSQL_PASSWORD

export SYMFONY__TEST_DATABASE_NAME=test

./setup_test_env.sh

Last but not least add the behat command to the test commands:

bin/behat -n

1	bin/behat -n

Things should be working now. Quickly enough you will run into the infamous xdebug “Fatal error: Maximum function nesting level of ‘100’ reached” error. Let’s fix this right away and add this in your setup commands:

echo "xdebug.max_nesting_level=200" >> /home/rof/.phpenv/versions/5.5/etc/php.ini

1	echo "xdebug.max_nesting_level=200" >> /home/rof/.phpenv/versions/5.5/etc/php.ini

Summary

So the complete setup commands dialog for phpspec and Behat together looks like this:

phpenv local 5.5
# Install dependencies through Composer
composer install --prefer-source --no-interaction
# Fix MySQL parameters
export SYMFONY__TEST_DATABASE_USER=$MYSQL_USER
export SYMFONY__TEST_DATABASE_PASSWORD=$MYSQL_PASSWORD
export SYMFONY__TEST_DATABASE_NAME=test
echo "xdebug.max_nesting_level=200" >> /home/rof/.phpenv/versions/5.5/etc/php.ini
./setup_test_env.sh

phpenv local 5.5

# Install dependencies through Composer

composer install --prefer-source --no-interaction

# Fix MySQL parameters

export SYMFONY__TEST_DATABASE_USER=$MYSQL_USER

export SYMFONY__TEST_DATABASE_PASSWORD=$MYSQL_PASSWORD

export SYMFONY__TEST_DATABASE_NAME=test

echo "xdebug.max_nesting_level=200" >> /home/rof/.phpenv/versions/5.5/etc/php.ini

./setup_test_env.sh

And the test commands like this:

bin/phpspec run --format=pretty --no-code-generation
bin/behat -n

1 2	bin/phpspec run --format=pretty --no-code-generation bin/behat -n

Everything should be working fine now! To run your tests local don’t forget to first execute the bash script (notice the extra dot, it is required):

. ./setup_test_env.sh

1	. ./setup_test_env.sh

Happy testing! 😉

Posted on October 31, 2014January 14, 2017 by ricbra 11

Ricbra's Blog

Loving the web since 2001

Ricbra's Blog

Tag: Behat

Speedup your test suite on Codeship using ParallelCI

Example configuration

Install Selenium headless on Debian Wheezy (optionally with Ansible)

How to use Codeship with Symfony2, phpspec and Behat

phpspec

Behat

Summary