04/2/13

ZF2 config caching with Composer

I recently stumbled upon a couple of config options within the Zend Framework 2 Skeleton Application, in the ./config/application.config.php file that relate to config caching. The options are:

<?php
return array(
    // These are various options for the listeners attached to the ModuleManager
    'module_listener_options' => array(
        // Whether or not to enable a configuration cache.
        // If enabled, the merged configuration will be cached and used in
        // subsequent requests.
        'config_cache_enabled' => $booleanValue,

        // The key used to create the configuration cache file name.
        'config_cache_key' => $stringKey,

        // The path in which to cache merged configuration.
        'cache_dir' => $stringPath,
    ),
);

First, some background: ZF2 uses multiple config files which it merges together at run-time to build the application config. This allows you to keep your config broken up into different logical components, and provides an easy way to have environment-specific (local) config files. However, this also increases resource usage and lowers performance, since it needs to read multiple files and merge multiple complex arrays each time it runs the application.

Luckily, the options I stumbled upon are designed to help! Enabling the config caching options tells the application to save the merged config in a cache file. This replaces multiple file reads, and lots of processing, with a single file read. Enabling caching is a simple matter of setting config\_cache\_enabled to true, and specifying config\_cache\_key and cache_dir.

For example:

<?php
return array(
    // These are various options for the listeners attached to the ModuleManager
    'module_listener_options' => array(
        // Whether or not to enable a configuration cache.
        // If enabled, the merged configuration will be cached and used in
        // subsequent requests.
        'config_cache_enabled' => true,

        // The key used to create the configuration cache file name.
        'config_cache_key' => 'my_project',

        // The path in which to cache merged configuration.
        'cache_dir' => __dir__ .'/../data/cache',
    ),
);

However, there is a catch… Since it is caching the config files, any time you make changes to the config you need to remove the cache file before the application will notice it. During production this isn’t too much of a problem, since you can remember to clear the cache when you roll out a new version, but during development it’s quite annoying. Now, ZF2 comes with the facility to manage local and global config, but this doesn’t apply to the ./config/application.config.php file, which is global by default, and leaves with you with having to remember to edit the config file in production (or development)… Which is something a lazy/busy developer doesn’t really want to have to remember.

So, the problem we face is that we can enable config caching easily, but as an all-or-nothing option. Is there a way to automatically enable it for production (and clear the cache whenever we perform a code update), but leave it disabled in development?

With the help of our good friend Composer: YES!

Composer has a fantastic scripts option that can be set within your application’s composer.json file. We can use this, and a little sed magic, to automatically toggle caching on and off depending on us being in production and development mode. A basic example looks like this:

{
    "scripts": {
        "post-install-cmd": [
            "sed -i \"s/'config_cache_enabled' => false,/'config_cache_enabled' => true,/g\" ./config/application.config.php",
            "rm -f ./data/cache/module-config-cache.my_project.php"
        ],
        "post-update-cmd": [
            "sed -i \"s/'config_cache_enabled' => true,/'config_cache_enabled' => false,/g\" ./config/application.config.php",
            "rm -f ./data/cache/module-config-cache.my_project.php"
        ]
    }
}

Let’s break it down:

post-install-cmd tells composer what commands to run after a successful execution of composer.phar install. Since you generally run this command when you install, and update, a production application, it will trigger the two commands in production. The first command uses sed to replace 'config_cache_enabled' => false, with 'config_cache_enabled' => true, within the application config, which as you should be able to tell will enable caching. The second command will remove any existing config cache file, to ensure a fresh version of the config is cached on the next page load.

post-update-cmd tells composer what commands to run after a successful execution of composer.phar update. This command is generally run on a development installation, since you usually want the latest and greatest dependencies for testing and dev. This does the opposite of the post-install-cmd, i.e. it disables the cache setting and removes the cache file just in case one exists.

To get all this working together, I configure the cache options but leave it disabled in git and have the scripts configured in my composer.json. This means that any time it changes in development, a quick git status will let me know, and as each time I roll out updates in production I also run composer.phar install, it will automatically be enabled.

That’s basically it :)

With a very small amount of setup, you now have config caching automatically enabling and disabling in ZF2 via Composer.

12/13/12

Loading ZF2 dev modules only when found

ZF2 comes with a new module system which makes it very easy to include 3rd party modules in your application. It works out of the box with Composer which makes it super easy.

One problem though is if you want to use a ZF2 module in development only, as there are two apparent options:

  1. Manually add it to your './config/application.config.php' file in development.
    OR
  2. Install it in production anyway.

But there is another way:

 array(
        'Application',
        // ...
    ),
    'module_listener_options' => array(
        'config_glob_paths'    => array(
            'config/autoload/{,*.}{global,local}.php',
        ),
        'module_paths' => array(
            './module',
            './vendor',
        ),
    ),
);

$devModules = Array(
    'ZFTool' => getcwd()."/vendor/zendframework/zftool/Module.php",
);

foreach ($devModules as $name => $class) {
    if (file_exists($class)) {
        $config['modules'][] = $name;
    }
}

return $config;
11/30/12

Automated backups of an on-demand MySQL database using automysqlbackup

Since I’m a PHP developer and I run Ubuntu on my laptop, I find it very easy to development directly on the machine. This means that my laptop also has PHP and MySQL running (and until the new PHP 5.4 dev server, Apache). While PHP doesn’t is only used on-demand, the MySQL server is a daemon that’s always running. Since I use my laptop for plenty of things other than dev with MySQL, I wanted a way to disable the autostart so I just need to start MySQL when I’m ready to use it.

A quick google found a useful answer on superuser.com:

To prevent mysql from starting on boot

  1. go to the /etc/init directory
  2. open the mysql.conf file
  3. comment out the “start on” line near the top of the file, the “start on” might be spread across two lines, so comment out both

If you want to manually start mysql, use the following command -

service mysql start

Which solves the first problem nicely, but how about backups?

I use a fantastic little script called automysqlbackup which for basic MySQL backup and versioning needs is perfect. It makes a database dump daily, saving a weekly and monthy backup periodically. So you end up with backups for the last 7 days, weekly backups for the last 5 weeks, and monthly backups are kept forever.

Installation on Ubuntu (or any Debian based system) is a piece of cake:

sudo apt-get install automysqlbackup

Once installed, it will start working automatically and save your backups into:

/var/lib/automysqlbackup

Configuration, which has quite sensible defaults, is managed through the config file:

sudo vim /etc/default/automysqlbackup

For a normal setup, you can leave it there, however, my MySQL database is no longer always online. This means that automysqlbackup is likely to attempt to backup when it is offline… and fail. So we have to get clever. I want to check if the MySQL database is online before running the backup. If it is offline, then we need to start it and save a tmp file. Once the backup is finished, check for the tmp file and shut down the database if found. This should result in automysqlbackup starting MySQL, running the backup, and stopping MySQL automatically when MySQL is offline, and just doing a backup when it’s online.

I wrote two little scripts to handle the pre-backup and post-backup tasks:

valorin@gandalf:~/mysql$ cat pre-backup.sh 
#!/bin/bash
# Remove left over tmp file
if [ -f /tmp/mysql-started-for-backup.tmp ]; then
    rm /tmp/mysql-started-for-backup.tmp
fi

# Start MySQL if offline
if [ -z "$(pgrep -u mysql -f /usr/sbin/mysqld)" ]; then
    service mysql start
    touch /tmp/mysql-started-for-backup.tmp
fi

valorin@gandalf:~/mysql$ cat post-backup.sh 
#!/bin/bash
# Some magic to get 'service mysql stop' to work
PATH=/usr/sbin:/usr/bin:/sbin:/bin

# Check for tmp file
if [ -f /tmp/mysql-started-for-backup.tmp ]; then
    rm /tmp/mysql-started-for-backup.tmp
    service mysql stop
fi

Automysqlbackup provides PREBACKUP and POSTBACKUP commands in the config file. My first assumption was to use the PREBACKUP script to check if MySQL is online, and if not, start it. But there is a problem… near the top of the configuration file is this line:

DBNAMES=`mysql --defaults-file=/etc/mysql/debian.cnf --execute="SHOW DATABASES" | awk '{print $1}' | grep -v ^Database$ | grep -v ^mysql$ | grep -v ^performance_schema$ | grep -v ^information_schema$ | tr \\\r\\\n ,\ `

Which attempts to connect to the MySQL database… so we need to inject our PREBACKUP script before this line is reached. To get around this, we can call the PREBACKUP script from the top of the config file, above the DBNAMES line:

...
# Command to run before backups 
/var/lib/automysqlbackup/pre-backup.sh
...
DBNAMES=`mysql --defaults-file=/etc/mysql/debian.cnf --execute="SHOW DATABASES" | awk '{print $1}' | grep -v ^Database$ | grep -v ^mysql$ | grep -v ^performance_schema$ | grep -v ^information_schema$ | tr \\\r\\\n ,\ `

But we can use the POSTBACKUP command to finish up the job since there are no more MySQL calls after it in the script:

# Command run after backups (uncomment to use)
POSTBACKUP="/var/lib/automysqlbackup/post-backup.sh"

There we have it – automated backups of an on-demand MySQL database using automysqlbackup. If you have a different way to handle this situation, I’d love to hear it!

11/21/12

How to inject a CSRF token into an XAJAX project

I’m working on a legacy project, adding in CSRF token support. One big problem I just encountered is that XAJAX (and YUI2) is used heavily in the project, and going through and editing every XAJAX request is painful and very time consuming…

What I was after is a simple global solution which will just work and then keep working without everyone having to remember to put in the CSRF token each time. I did some googling and found a helpful article that explains how to do it via request headers in jQuery (Automatically adding CSRF tokens to ajax calls when using jQuery), but since the project I’m working on is using XAJAX and YUI2 I couldn’t use it as-is.

After some digging and mining in the Chrome JS Inspector, I found a very simple solution.

Simply add this configuration line into the <head>...</head> section of your project with the rest of your XAJAX config, replacing YOUR_CSRF_TOKEN with the current user’s CSRF token on page generation.

<script type="text/javascript">
    xajax.config.postHeaders = {'X-CSRFToken': 'YOUR_CSRF_TOKEN'};
</script>

Pretty simple really :)

10/31/12

How to automatically create testing databases (in PHP & MySQL)

If you use an automated testing platform (such as Jenkins) you may have come across the problem where you need to manually create a new database (and sometimes user) every time you configure a different build or a different host. For small projects it’s not much of a chore, but for larger projects (or complicated build environments) it quickly becomes a pain.

To try and get around this problem, you need to do two things:

  1. Create a database user who has permissions to create new databases.
  2. Create a script that checks for a config file, and creates a database and new config if required.

Since I’m a PHP+MySQL guy, this will be specifically based on those two. But the concepts can be ported to any language and database.

#1 – Creating a database user

For security reasons, the user in #1 should only have access to create databases with a prefix. This will ensure that if those database credentials are exposed, the damage will be minimal and the testing databases can be easily identified. To do this in MySQL you can restrict permissions to a database name with wildcard.

In this case, we want to create a new database user project with the password password, along with setting the permissions to create database tables with the prefix project_. The MySQL looks like:

CREATE USER 'project'@'localhost' IDENTIFIED BY 'password';
CREATE USER 'project'@'%'         IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON `project\_%`.* TO 'project'@'localhost';
GRANT ALL PRIVILEGES ON `project\_%`.* TO 'project'@'%';

FLUSH PRIVILEGES;

#2 – Check for test database script

In ZF2 your database credentials are stored in ./config/autoload/local.php, which shouldn’t be included in your version control. If you use something other than ZF2 that doesn’t have a local config file, you will have to get creative :)

We can take advantage of a local config file by checking if the file exists. If the file does not exist, we can assume that we need to create a new database, and go ahead and do it. Finally we need to create the config file so it can be used by the application.

My finished script looks like:

#!/usr/bin/php
 'Mysqli',
        'hostname' => $hostname,
        'username' => $username,
        'password' => $password,
));

// Create a new database with random name
$database = $prefix.mt_rand(10000, 99999);
$adapter->query(
    "CREATE DATABASE IF NOT EXISTS `{$database}` DEFAULT CHARACTER SET utf8",
    Zend\Db\Adapter\Adapter::QUERY_MODE_EXECUTE
);

// Write config file
$config = << array(
        'dsn'      => 'mysql:dbname={$database};host={$hostname}',
        'username' => '{$username}',
        'password' => '{$password}',
    ),
);

EOF;

file_put_contents($file, $config);
echo "\nConfig file created for database: {$database}.\n\n";

Pretty simple really :)