Introduction

Upgrading a ProcessMaker installation requires, in addition to replacing the application code, the execution of the following command:

cd /opt/processmaker
php ./processmaker upgrade

Executing the lines above will cause ProcessMaker to upgrade ALL existing workspaces, which, in most cases, is the desired behavior. However, under certain circumstances, that approach is unacceptable. In particular, upgrading the Enterprise Trial server, which has several thousand workspaces, takes too long to run, given the fact that workspaces are upgraded sequentially (i.e. one after the other) instead of in parallel, and that each workspace takes several minutes to upgrade. Therefore, an alternative method was required. The most obvious one is to issue multiple parallel, per-workspace upgrade commands (for example, under different ssh sessions on the server). This approach was tested  but resulted in race conditions that broke XML translation files, as well as the contents of translation tables in the database.

Due to the previous challenges, specially the inability to upgrade several workspaces in parallel, it was decided that the best approach was to only upgrade workspaces on-demand, whenever a user logged in and it was found that his workspace was not upgraded yet. The approach had to consider the scenario in which two users with non-upgraded workspaces tried to log in at the same time, and avoid to upgrade both workspaces in paralle, but instead, upgrade one after the other.

This page describes the mechanism implemented to achieve on-demand upgrade of ETrial workspaces avoiding race conditions.

Solution

Overview

The implemented solution is described in a general fashion below:

1. User login attempts are intercepted. The workspace is checked to determine whether it has already been upgraded or not. In case it has, login continues normally. Otherwise. go to step 2.
2. A Linux at job is scheduled to run 1 minute after the intercepted login attempt. The exact command to be run is the upgrade of the user's workspace: php processmaker upgrade workspacename. Please notice that this needs the atd linux daemon to be running.
3. Once the at job is scheduled, the user is redirected to an upgrade status page which includes a progress bar. The progress bar moves from 0 to 100 as the corresponding upgrade log file number of lines increases.
4. Once the upgrade is completed, the upgrade status page shows a Success message and a link to log into processMaker.

Intercepting user logins

To intercept users logging in, the core code of ProcessMaker was modified. Specifically, the login/authentication method was modified by adding an include to a custom interceptor.php file, as shown below:

<?php
/**
...
 * For more information, contact Colosa Inc, 2566 Le Jeune Rd.,
 * Coral Gables, FL, 33134, USA, or email info@colosa.com.
 *
 */
try {
include_once('interceptor.php');
    $usr = '';
    $pwd = '';
...

Determining whether an upgrade is required and scheduling it

The interceptor.php file verifies whether the user's workspace has been upgraded or not and, if not, schedules the at job.

<?php
$oConnection = Propel::getConnection('workflow');
$sql = "SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='EMAIL_EVENT' AND column_name='EMAIL_SERVER_UID' AND table_schema = 'wf_".SYS_SYS."'";
$oStatement  = $oConnection->prepareStatement($sql);
$res = $oStatement->executeQuery();
if( $res->getRecordCount() < 1 ) {
  var_dump($res->getRecordCount());
  $script =  "/usr/bin/php /opt/pmupgrader.php";
  $cmd1 = "echo ".$script." ".SYS_SYS;
  $cmd2 = "at now + 1 minutes";
  $redirect = " 1>> /tmp/".SYS_SYS."_pmupgrade_at.log 2>&1";
  $cmd = $cmd1." | ".$cmd2." ".$redirect;
  $resArr = array();
  $res = exec($cmd,$resArr);
  G::header("Location: /trialUpgradeSite/progressFile.php?ws=".SYS_SYS);
}
?>

The check on whether the workspace has been upgraded or not is based on verifying the existence of a certain column of a table (column EMAIL_SERVER_UID from table EMAIL_EVENT) that, at the time of upgrade from 3.0.1.8 to 3.1, was only present when the upgrade has already been performed. The command required to schedule the at job is assembled by using the workspace name and location of upgrade script. The actual command that is executed is the following:

echo /usr/bin/php /opt/pmupgrader.php workspaceName | at now + 1 minutes 1>> /tmp/workspaceName_pmupgrade_at.log 2>&1

Notice that output is redirected to a log file in /tmp so that it can be traced later on whether the job was successfully scheduled or not.

Upgrading the workspace

The actual upgrade is performed by the script at /opt/pmupgrader.php :

<?php

// Connection parameters
$host="localhost";
$user="ET3sales";
$pass="4wxz2mr2MRbaVHXB";
$db="wf_ejortegau";

ini_set( 'date.timezone', 'America/New_York' );

// Open general log
$general_log=fopen("/tmp/upgrader_general.log","a");

$mydb = new mysqli($host,$user,$pass,$db);
if ($mydb->connect_errno) {
    fprintf($general_log, "Connect failed: %s\n", $mydb->connect_error);
    fclose($general_log);
    die();
}

if(!isset($argv[1])) {
    fprintf($general_log, "No workspace specified\n");
    fclose($general_log);
    die();
}

$ws = $argv[1];
fprintf($general_log, "Will start upgrade of workspace ".$ws."\n");
$sql = "SELECT GET_LOCK('upgrade_in_progress',-1)";
fclose($general_log);
$ws_log = fopen("/tmp/".$ws."_pmupgrader.log","a");
fprintf($ws_log,"Starting upgrade of workspace ".$ws."\n");
fprintf($ws_log,"Attempting to acquire lock at ".date("Y-m-d H:i:s")."\n");
$res = $mydb->query($sql);
fprintf($ws_log,"Lock acquired at ".date("Y-m-d H:i:s")."\n");
$cmd="cd /opt/processmaker;php ./processmaker upgrade --no-xml ".$ws." > /tmp/".$ws."_processmaker_upgrade.log";
echo $cmd."\n";
echo exec($cmd);
$sql = "SELECT RELEASE_LOCK('upgrade_in_progress')";
fprintf($ws_log,"Attempting to release lock at ".date("Y-m-d H:i:s")."\n");
$res = $mydb->query($sql);
fprintf($ws_log,"Lock released at ".date("Y-m-d H:i:s")."\n");
$mydb->close();
fprintf($ws_log,"Finished upgrade of workspace ".$ws."\n");
?>

In order to avoid race conditions, the upgrade script establishes a connection to a MySQL database. The actual database being used is not relevant. What matters is that a named lock is acquired on that database before starting the upgrade, and released once it is completed. For details about MySQL named locks, refer to this documentation. Once the lock is acquired, the corresponding workspace upgrade command is executed (and its output redirected to a workspace-specific log file under /tmp/ via standard shell redirection), namely:

cd /opt/processmaker;php ./processmaker upgrade --no-xml ".$ws." > /tmp/".$ws."_processmaker_upgrade.log

In case two (or more) upgrades are attempted at the same time, the MySQL named lock ensures only one of them is executed at a time. Once the first upgrade is done, the lock is released and the second one can acquire it.

Upgrade status page

When a workspace upgrade is scheduled via the atd job, the user is redirected to a status page. This status page performs an AJAX call to a backend php script that checks the upgrade log file and, based on that, provides feedback to the user regarding whether their upgrade has started, what's its current (approximate) percentage completion, whether there were errors during its execution, and whether it has been completed successfully. The status page is comprised of several files that have been placed under /opt/processmaker/workflow/public_html/trialUpgradeSite/ . There are two main files there, namely:

• progressFile.php: this file has the actual content displayed to the user, and includes all JS and CSS files required, including the AJAX calls to countFileLines.php to obtain the current status of the upgrade as a percentage and message.
• countFileLines.php: this file is the backend AJAX script that checks upgrade status from the upgrade log file and based on that returns a percentage complete of a running upgrade, or a failed or success message depending on the output status. The percentage calculation is based on dividing the number of lines of the log file in the number of expected lines a full upgrade log file should have. Since this number may vary from installation to installation (and, in our experience, also from workspace to workspace), an educated guess about the maximum possible number of lines was made and hard-coded in the script.

Publishing the Upgrade status page

Since the pmos.conf file includes configuration for the rewrite engine that make sure that every request mapped to DocumentRoot (/opt/processmaker/workflow/public_html) are handled by the app.php script, additional configuration is required to ensure that the upgrade status page can be accessed. The actual configuration is based on adding an additional rewrite condition to the rewrite rule of the configuration file, to ensure that requests to the trialUpgradeSite subdirectory are not passed to the app.php script. Therefore, the pmos.conf file is changed as shown below:

...
        <IfModule mod_rewrite.c>
            RewriteEngine On
            RewriteCond %{REQUEST_FILENAME} !-f
            RewriteCond %{REQUEST_URI} !^trialUpgradeSite/progressFile.php
            RewriteRule ^.*/(.*)$ app.php [QSA,L,NC]
        </IfModule>
...