Chamilo : Optimization Guide

Documentation > Optimization Guide

In seldom cases, you will need to start looking into efficiency issues with Chamilo. This guide is a work in progress intended to help administrators optimize their Chamilo installation.

Contents

  1. Using xCache, APC, Memcache or Memcached
  2. Slow queries
  3. Indexes caching
  4. Sessions directories
  5. Users upload directories
  6. Zlib compressed output
  7. Memory considerations for high numbers of users
  8. Avoiding non-fixed values
  9. Speeding file downloads with mod_xsendfile
  10. IGBinary for faster courses backups and better sessions
  11. Removing files download permissions check
  12. MySQL/MariaDB compression

1. Using xCache or APC

See xCache's website for summary documentation.
Set your xcache.ini configuration (/etc/php5/conf.d/xcache.ini) to match your system. For example, you *could* have something like this (intentionally hiding comments here): 
xcache.shm_scheme =        "mmap"
xcache.size  =                32M
xcache.count =                 2
xcache.slots =                8K
xcache.ttl   =                 0
xcache.gc_interval =           0
xcache.var_size  =           16M
xcache.var_count =            16
xcache.var_slots =            8K
xcache.var_ttl   =            60
xcache.var_maxttl   =        300
xcache.var_gc_interval =     300
xcache.test =                Off
xCache will feel useless until you actually start to put some variables in cache. If you're showing the "Who is online" counter, that's one of the best item there is to implement xCache.
For example, you could implement it this way (in main/inc/lib/banner.lib.php):
$xc = function_exists('xcache_isset');
$number = 0;
if ((api_get_setting('showonline', 'world') == 'true' AND !$user_id) OR (api_get_setting('showonline', 'users') == 'true' AND $user_id) OR (api_get_setting('showonline', 'course') == 'true' AND $user_id AND $course_id)) {
  if ($xc && xcache_isset('campus_chamilo_org_whoisonline_count_simple')) {
    $number = xcache_get('campus_chamilo_org_whoisonline_count_simple');
  } else {
    $number = who_is_online_count(api_get_setting('time_limit_whosonline'));
    xcache_set('campus_chamilo_org_whoisonline_count_simple',$number);
  }
}
$number_online_in_course = 0;
if(!empty($_course['id'])) {
  if ($xc && xcache_isset('campus_chamilo_org_whoisonline_count_simple_'.$_course['id'])) {
    $number_online_in_course = xcache_get('campus_chamilo_org_whoisonline_count_simple_'.$_course['id']);
  } else {
    $number_online_in_course = who_is_online_in_this_course_count(api_get_user_id(), api_get_setting('time_limit_whosonline'), $_course['id']);
    xcache_set('campus_chamilo_org_whoisonline_count_simple_'.$_course['id'],$number_online_in_course);
  }
}
Note that, as xCache is a shared caching system, it is very important to prefix your variables with a domain name or some kind of identifier, otherwise it would end up in disaster if you use a shared server for several portals.
If you use php5-memcache, then this piece of code would look like this (you need to adjust depending on your settings): 
    global $_configuration;
    $_course    = api_get_course_info();
    $course_id  = api_get_course_id();
    $user_id    = api_get_user_id();

    $html = '';
    $xc = method_exists('Memcache','add');
    if ($xc) {
        // Make sure the server is available
        $xm = new Memcache;
        $xm->addServer('localhost', 11211);
        $xc = $xc && ($xm->getServerStatus('localhost',11211)!=0);
        // The following concatenates the name of the database + the id of the
        // access url to make it a unique variable prefix for the variables to
        // be stored
        $xs = $_configuration['main_database'].'_'.$_configuration['access_url'].'_';
    }
    $number = 0;
    if ((api_get_setting('showonline', 'world') == 'true' AND !$user_id) OR (api_get_setting('showonline', 'users') == 'true' AND $user_id) OR (api_get_setting('showonline', 'course') == 'true' AND $user_id AND $course_id)) {
        if ($xc && $xm->get($xs.'wio_count_simple')) {
            $number = $xm->get($xs.'wio_count_simple');
        } else {
            $number = who_is_online_count(api_get_setting('time_limit_whosonline'));
            $xm->set($xs.'wio_count_simple',$number,0,30);
        }

        $number_online_in_course = 0;
        if(!empty($_course['id'])) {
            if ($xc && $xm->get($xs.'wio_count_simple_'.$_course['id'])) {
                $number_online_in_course = $xm->get($xs.'wio_count_simple_'.$_course['id']);
            } else {
                $number_online_in_course = who_is_online_in_this_course_count($user_id, api_get_setting('time_limit_whosonline'), $_course['id']);
                $xm->set($xs.'wio_count_simple_'.$_course['id'],$number_online_in_course,0,30);
            }
        }

An optional additional caching mechanism you may use is the realpath_cache_size and realpath_cache_ttl php.ini parameters. See the PHP documentation for more details.

If you prefer using APC, you can use the same kind of trick as above, just changing the code a little: 
    $xc = function_exists('apc_exists');
    $number = 0;
    if ((api_get_setting('showonline', 'world') == 'true' AND !$user_id) OR (api_get_setting('showonline', 'users') == 'true' AND $user_id) OR (api_get_setting('showonline', 'course') == 'true' AND $user_id AND $course_id)) {
        if ($xc) {
            $apc = apc_cache_info(null,true);
            $apc_end = $apc['start_time']+$apc['ttl'];
            if (apc_exists('my_campus_whoisonline_count_simple') AND (time() < $apc_end) AND apc_fetch('my_campus_whoisonline_count_simple') > 0 ) {
                $number = apc_fetch('my_campus_whoisonline_count_simple');
            } else {
                $number = who_is_online_count(api_get_setting('time_limit_whosonline'));
                apc_clear_cache();
                apc_store('my_campus_whoisonline_count_simple',$number,15);
            }
        } else {
                $number = who_is_online_count(api_get_setting('time_limit_whosonline'));
        }
        $number_online_in_course = 0;
        if (!empty($_course['id'])) {
            if ($xc) {
                $apc = apc_cache_info(null,true);
                $apc_end = $apc['start_time']+$apc['ttl'];
                if (apc_exists('my_campus_whoisonline_count_simple_'.$_course['id']) AND (time() < $apc_end) AND apc_fetch('my_campus_whoisonline_count_simple_'.$_course['id']) > 0) {
                    $number_online_in_course = apc_fetch('my_campus_whoisonline_count_simple_'.$_course['id']);
                } else {
                    $number_online_in_course = who_is_online_in_this_course_count($user_id, api_get_setting('time_limit_whosonline'), $_course['id']);
                    apc_store('my_campus_whoisonline_count_simple_'.$_course['id'],$number_online_in_course,15);
                }
            } else {
                $number_online_in_course = who_is_online_in_this_course_count($user_id, api_get_setting('time_limit_whosonline'), $_course['id']);
            }
        }
     ...

If you use php5-memcached (different set of functions than php5-memcache!), then this piece of code would look like this (you need to adjust depending on your settings): 
    global $_configuration;
    $_course    = api_get_course_info();
    $course_id  = api_get_course_id();
    $user_id    = api_get_user_id();

    $html = '';
    $xc = method_exists('Memcached', 'add');
    if ($xc) {
        // Make sure the server is available
        $xm = new Memcached;
        $xm->addServer('localhost', 11211);
        // The following concatenates the name of the database + the id of the
        // access url to make it a unique variable prefix for the variables to
        // be stored
        $xs = $_configuration['main_database'].'_'.$_configuration['access_url'].'_';
    }
    $number = 0;
    if ((api_get_setting('showonline', 'world') == 'true' AND !$user_id) OR (api_get_setting('showonline', 'users') == 'true' AND $user_id) OR (api_get_setting('showonline', 'course') == 'true' AND $user_id AND $course_id)) {
        if ($xc) {
            if ($xm->get($xs.'wio_count_simple')) {
                $number = $xm->get($xs.'wio_count_simple');
            } else {
                $number = who_is_online_count(api_get_setting('time_limit_whosonline'));
                $xm->set($xs.'wio_count_simple',$number,120);
            }
        } else {
              $number = who_is_online_count(api_get_setting('time_limit_whosonline'));
        }
        $number_online_in_course = 0;
        if (!empty($_course['id'])) {
            if ($xc) {
                if ($xm->get($xs.'wio_count_simple_'.$_course['id'])) {
                    $number_online_in_course = $xm->get($xs.'wio_count_simple_'.$_course['id']);
                } else {
                    $number_online_in_course = who_is_online_in_this_course_count($user_id, api_get_setting('time_limit_whosonline'), $_course['id']);
                    $xm->set($xs.'wio_count_simple_'.$_course['id'],$number_online_in_course,120);
                }
            } else {
                $number_online_in_course = who_is_online_in_this_course_count(api_get_user_id(), api_get_setting('time_limit_whosonline'), $_course['id']);
            }
        }
        // ...

It is also worth noting that the Université de Genève, Switzerland, observed that the calculation of the total size used by course documents is one of the heaviest queries in Chamilo, so you might want to cache the results of this one as well, using the same technique.

Finally, if your portal is highly public *and* you are showing the popular courses on the homepage, you might want to also reduce the amount of queries this generates, using the same technique as above, but for the main/inc/lib/auth.lib.php library, looking for the "Tracking::get_course_connections_count()" call:

        while ($row = Database::fetch_array($result)) {
            $row['registration_code'] = !empty($row['registration_code']);
            $count_users = CourseManager::get_users_count_in_course($row['code']);
            $xc = function_exists('apc_exists');
            if ($xc) {
                $apc = apc_cache_info(null, true);
                $apx_end = $apc['start_time']+$apx['ttl'];
                if (apc_exists('my_campus_course_visits_'.$row['code']) AND (time() < $apc_end) AND apc_fetch('my_campus_course_visits_'.$row['code']) > 0) {
                    $count_connections_last_month = apc_fetch('my_campus_course_visits_'.$row['code']);
                } else {
                    $count_connections_last_month = Tracking::get_course_connections_count($row['code'], 0, api_get_utc_datetime(time() - (30 * 86400)));
                    apc_store('my_campus_course_visits_'.$row['code'], $count_connections_last_month, $apc['ttl']);
                }
            } else {
                $count_connections_last_month = Tracking::get_course_connections_count($row['code'], 0, api_get_utc_datetime(time() - (30 * 86400)));
            }
            ...
        }
Finally, the Free Campus of Chamilo has a very specific case of slow query: the courses catalog! Because there might be more than 30,000 courses in there, getting the number of "Connections last month" can be a desastrous query in terms of performances. This is why you should try to cache the results as well.
Obviously, as we are speaking about showing the number of visits this month, it doesn't really matter if the number doesn't refresh for an hour or so...
Locate the main/inc/lib/course_category.lib.php file, open it and go to the browseCoursesInCategory() function.
Locate the $count_connections_last_month = Tracking::get_course_connections_count(...) call, and wrap in into something like this: 
    $xc = method_exists('Memcached', 'add');
    if ($xc) {
        // Make sure the server is available
        $xm = new Memcached;
        $xm->addServer('localhost', 11211);
        // The following concatenates the name of the database + the id of the
        // access url to make it a unique variable prefix for the variables to
        // be stored
        $xs = $_configuration['main_database'].'_'.$_configuration['access_url'].'_';
    }
    $result = Database::query($sql);
    $courses = array();
    while ($row = Database::fetch_array($result)) {
        $row['registration_code'] = !empty($row['registration_code']);
        $count_users = CourseManager::get_users_count_in_course($row['code']);
        if ($xc) {
            if ($xm->get($xs.'cccount_'.$row['code'])) {
                $number = $xm->get($xs.'cccount_'.$row['code']);
            } else {
                $count_connections_last_month = Tracking::get_course_connections_count($row['code'], 0, api_get_utc_datetime(time() - (30 * 86400)));
                $xm->set($xs.'cccount_'.$row['code'], $count_connections_last_month, 3600);
            }
        } else {
            $count_connections_last_month = Tracking::get_course_connections_count($row['code'], 0, api_get_utc_datetime(time() - (30 * 86400)));
        }
   ...

2. Slow queries

Enable slow_queries in /etc/mysqld/my.cnf, restart MySQL then follow using sudo tail -f /var/log/mysql/mysql-slow.log

In Chamilo 1.9 in particular, due to the merge of all databases into one, you might experience performance issue if you have many learning paths with many items in them.
To solve this performance issue, you can execute the following queries manually in your database:
ALTER TABLE lp_item ADD INDEX idx_c_lp_item_cid_lp_id (c_id, lp_id);
ALTER TABLE lp_item_view ADD INDEX  idx_c_lp_item_view_cid_lp_view_id_lp_item_id (c_id, lp_view_id, lp_item_id);
ALTER TABLE user_rel_tag ADD INDEX idx_user_rel_tag_user (user_id);
In Chamilo 1.9.8, we use the c_item_property table more actively. This causes issues with the reporting pages for the assignments. You can reduce the impact by adding the following index: 
alter table c_item_property add index idx_itemprop_tooliuid(tool, insert_user_id);
These will be available in Chamilo 1.10 directly, but we cannot put them into Chamilo 1.9 from now on for organizational reasons.

3. Indexes caching

One good reference: MySQL documentation on multiple key caches

4. Sessions directories

php_admin_value session.save_path 1;/var/www/test.chamilo.org/sessions/

5. Users upload directories

Create 10 directories inside the app/upload/users directory (from 0 to 9) and update your admin settings. This has to be done at install & configuration time, otherwise you might loose user data (or have to write a script for data distribution).

6. Zlib compressed output

Although this will not make your server faster, compressing the pages you are sending to the users will definitely make them feel like your website's responses are a lot faster, and thus increase their well-being when using Chamilo.

Zlib output compression has to be set at two levels: PHP configuration for PHP pages and Apache for images and CSS.

To update the PHP configuration (either in php.ini or in your VirtualHost), use the zlib.output_compression. If you set this inside your Apache's VirtualHost, you should use the following syntax. 
php_value zlib.output_compression 1

Configuring your Apache server to use output compression is a bit trickier. You have to use the mod_deflate module to do it. Your configuration should look like something like this (please read the corresponding documentation before implementing in production).
Easy mode: 
AddOutputFilterByType DEFLATE text/html text/plain text/xml
or, for every content type (dangerous) you can put the following inside a location or directory block:
SetOutputFilter DEFLATE

Advanced mode: 

# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
Don't have time or resources to optimize your Chamilo installation yourself? Hire an official Chamilo provider and get it sorted out professionally by specialists. Valid XHTML 1.0 Transitional Valid CSS

Memory considerations for high numbers of users

Some administration scripts *have to* handle lists of all users, and this might have a considerable impact on portals with very high numbers of users. For example, the main/admin/add_users_to_session.php script that handles the registration of users into a specific session, if used with the (non-default) full list of users, will devour about 3KB per user, which, for 100,000 users, translates into the need for around 300MB of RAM just to show this page, and to around 3GB for 1,000,000 users.
This mode is not loaded by default, but could still be selected, leading to a "Fatal error: Allowed memory size ... exhausted" message.
The only non-scripted solution here is to allow for the corresponding amount of RAM for your PHP configuration (memory_limit = 300M) or your specific VirtualHost if you use mod-php5 (php_value memory_limit 300M).

Avoiding non-fixed values

Many things in Chamilo are written focusing on the ease of use, even for the administrator. Sometimes, these settings are weighing a little bit more on the system. This is the case, between others, of the mail.conf.php file (being loaded unconditionally) and its CONSTANT "IS_WINDOWS_OS", which is defined by a function call (api_is_windows_os()) at the beginning of main_api.lib.php. The definition of this constant (which is executed at *every* page load) can easily be avoided, and the only place where it is used inconditionally (mail.conf.php) can be modified to set the line as you expect it (depending on whether you use sendmail/exim or smtp). 
$platform_email['SMTP_MAILER']       = 'smtp';
or 
$platform_email['SMTP_MAILER']       = 'mail';
In fact, the complete loading of mail.conf.php can also be avoided if loaded conditionally (with require_once) when sending an e-mail (which is the only case where it is useful).

As an additional node, on very active portals with a lot of courses for each users, the icons that appear next to the courses illustrating changes in the corresponding course might be heavyweighted. You can alter slightly the behaviour by not querying for notifications you don't care about, like dropbox, notebook or chat. Change this in main/inc/lib/display.lib.php, in function show_notification().

Speeding file downloads with mod_xsendfile

It might have come to your attention that file downloads through Chamilo might get slow, under default conditions, in particular using Apache 2.

There are several ways to fix this, one of which is removing the .htaccess inside the courses/ directory. This, however, will remove all permissions checks on the files contained in this directory, so... most of the time, not ideal unless your portal is *really* open to the world.

Another technique, revealed to us by VirtualBlackFox on this Stackoverflow post, is to use the X-SendFile module for Apache 2.2+ (other web servers might offer other solutions, or avoid the problem initially).

Installing the X-SendFile module will depend on your operating system, but if you use Ubuntu, you'll have to check you are including the "universe" repository inside your packages sources (check /etc/apt/sources.list), then: 

sudo apt-get update
sudo apt-get install libapache2-mod-xsendfile
sudo service apache2 restart
Once you're done with installing, you'll have to configure Chamilo to use it.
First, edit your VirtualHost or your Apache configuration in general (in Ubuntu, check the /etc/apache2/ or /etc/apache2/sites-available/ folder). This is done by adding the following line inside your configuration, and reloading Apache (example provided on the basis of a virtual host located in /etc/apache2/sites-available/my.chamilo.net.conf) : 
sudo vim /etc/apache2/sites-available/my.chamilo.net.conf
# add the following line:
  X-SendFile on
# exit the file
sudo service apache2 reload
Finally, you'll have to got to your Chamilo configuration file, and add the following line at the very bottom of the file main/inc/conf/configuration.php: 
$_configuration['enable_x_sendfile_headers'] = true;
Done! Now your downloads should go substantially faster. This is still a feature in observation. We're not sure the benefits are sufficient, so don't hesitate to let us know in the related issue in Chamilo's tracking system

IGBinary for courses backups and better sessions management

IGBinary is a small PECL library that replaces the PHP serializer. It uses less space (so less memory for serialized objects) and is particularly efficient with memory-based storages (like Memcached). Use it for course backups (see issue 4443) or to boost sessions management.

Removing files download permissions check

This measure is not cumulative with mod_xsendfile explained above. It is not *recommended* either, as it removes an important security layer.

In Chamilo, for security and tracking purposes, all downloaded files pass through PHP scripts that check whether the user has access to the file given his/her current permissions. This process requires important database accesses and processing, which might terminally affect your server's performance. In particular, this can have a huge effect if having hundreds of simultaneous users accessing learning paths pages composed of local resources.

The logic behind this verification is that, whatever resources that needs to be downloaded/viewed that come from the /courses/ directory, the /courses/.htaccess file with get in the middle and redirect these accesses to a PHP script (usually called download.php but there are more than one depending on the type of resource).

If you want to speed up files accesses and you don't really care about whom can see your files, then an option is to simply remove this redirection to download.php and let Apache treat the file on its own.

Furthermore, using a PHP script for the download (unless you have special rules) will usually prevent static content caching, which will multiply downloads and use large amount of additional bandwidth.

Typically, the .htaccess will look like this (with additional comments):


RewriteEngine On
RewriteBase /courses/
RewriteCond %{REQUEST_URI} !^/main/
RewriteRule ([^/]+)/document/(.*)&(.*)$ $1/document/$2///$3 [N]
RewriteRule ([^/]+)/scorm/(.*)$ /main/document/download_scorm.php?doc_url=/$2&cDir=$1 [QSA,L]
RewriteRule ([^/]+)/document/(.*)$ /main/document/download.php?doc_url=/$2&cDir=$1 [QSA,L]
RewriteRule ([^/]+)/work/(.*)$ /main/work/download.php?file=work/$2&cDir=$1 [QSA,L]

The idea is to remove the "scorm" line, so you are left with: 

RewriteEngine On
RewriteBase /courses/
RewriteCond %{REQUEST_URI} !^/main/
RewriteRule ([^/]+)/document/(.*)&(.*)$ $1/document/$2///$3 [N]
RewriteRule ([^/]+)/document/(.*)$ /main/document/download.php?doc_url=/$2&cDir=$1 [QSA,L]
RewriteRule ([^/]+)/work/(.*)$ /main/work/download.php?file=work/$2&cDir=$1 [QSA,L]

This is easy, doesn't require a server reload and you should see the results pretty quickly. As mentioned above, if security of your content is an issue, though, you should avoid using this technique.

You can also mitigate the risk by disabling permissions check only for some static resource like css,js and fonts files.
For that is required to load header module in apache (check with a2enmod in your favorite root terminal)
add theses line after RewriteBase /courses/: 


    # all file name ended with these  extensions names will bypass the permission check   (and also served by the browser cache at  the next request)
    
        Header unset Cache-Control  
        Header set Cache-Control "public, max-age=29030400" 
        RequestHeader unset Cookie      
        Header unset ETag

MySQL/MariaDB compression

If your database server is separate from your web server, you have to play with bandwidth, firewalls, and network restrictions in general.
In particular, when dealing with large-scale portals, the time a SQL query will take to return to the web server will take longer and, eventually, in the most critical cases, will take too long, and your web servers will be completely overloaded (load average very high because the system is waiting for I/O operations, but processors usage not being very high is a clear sign of this).
To solve this kind of issues, MySQL and MariaDB offer a data compression mechanism, which will reduce the amount of data passed between PHP and the database server. Ultimately, this reduction will lower bandwidth usage and reduce the impact of numerous and heavy data requests (and save you).
In 1.10.0, we have added the possibility to enable this compression very easily, from the configuration.php file, uncommenting the following line: 

//$_configuration['db_client_flags'] = MYSQL_CLIENT_COMPRESS;
This should have an immediate effect on the load average on your server.

Authors

Don't have time or resources to optimize your Chamilo installation yourself? Hire an official Chamilo provider and get it sorted out professionally by specialists. Valid XHTML 1.0 Transitional Valid CSS