Home » PHP » TBG - The BugGenie performance analyze

TBG - The BugGenie performance analyze

Published on 03.11.2011, by Lubos Dzurik

This article will be removed once TBG 3.2 released.

This is a shortlist of tips to improve the performance of issue tracking system and project management tool http://www.thebuggenie.com/ (hereinafter "TBG").

This analyse has been done by my colleague developer Vlado and myself using profiling tools and debugging the code.

The reason why we did this is, that we truly appreciate effort invested into the tracker that has good potential. On the other hand we jumped into some issues while setting up TBG 3.1.4 so we took a deeper look into it and found some more issues...

So, here we go - few tips on how to improve TBG general performance.

Tip 1 - Optimize caching the table schemas

Database schemas (70+ tables) cached under [cache/B2DB/*.*] should all be contained within a single file only once. Caching separately each table puts heavy load on filesystem - on each request tens of files must be loaded.

Even worse thing is that at the end of each request all cached files are written back under [cache/B2DB/*.*]. This is high load put on filesystem, not mentioning exclusive file locks disabling access to the file while being saved on harddisk for paralel requests.

Even the best hosting cannot handle badly designed cache, right?

Solution:
Parse all database table schemas at once at first request into a single file. Then use this file only for reading.

Tip 2 - Do not gzip contents streamed down to browser (CSS, JS)

This may compromise server settings and/or proxy servers. Leave gzip compression to server itself. Also Internet Explorer (8) won't load CSS files and javascripts.

Solution:

Let PHP combine CSS and Javascript files into a single file. That's it - then just output it without compression.

Optionally provide .htaccess within webroot along with the index.php file enabled compression like this:

AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript

# http://littletechthoughts.blogspot.com/2009/02/invalid-command-header-perhaps.html
# apache must have enabled mod_headers + mod_deflate




	# Insert filter               
	SetOutputFilter DEFLATE          

	# Netscape 4.x has some problems...               
	BrowserMatch ^Mozilla/4 gzip-only-text/html               

	# Netscape 4.06-4.08 have some more problems               
	BrowserMatch ^Mozilla/4\.0[678] no-gzip               

	# MSIE masquerades as Netscape, but it is fine               
	# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html               

	# NOTE: Due to a bug in mod_setenvif up to 
	# Apache 2.0.48 the above regex won't work.
	# You can use the following               
	# workaround to get the desired effect:               
	BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html               

	# Don't compress images               
	SetEnvIfNoCase Request_URI \               
	\.(?:gif|jpe?g|png)$ no-gzip dont-vary               

	# Make sure proxies don't deliver the wrong content               
	# this requires mod_headers!               
	Header append Vary User-Agent env=!dont-vary

Tip 3 - Never rely upon external resources

Assume TBG will be installed in intranet with forbidden access to internet due to security. That's what big companies do - network admnistrators gets paranoid and they are allowed to legally live with this disorder:-) All resources should be self contained within TBG installation.

The only exception is of course authentication via Openid or other external mechanism, but never for generated contents (jquery, images, gravatar should not be loaded from various servers).

I calculated that in average each TBG request loads resources from 1-3 external servers (fonts.googleapis.com, themes.googleusercontent.com, www.gravatar.com).

Tip 4 - Add indexes to the database joined columns

It is a MUST for RDBMS to optimize related columns through adding indexes on joined and heavily used columns. As an example of great database optimization is database schema for PHPBB3 forum. Pretty complex relations with excellent performance - well indexed database.

Tip 5 - Any reason for using JQuery and Prototype together?

Only one framework should be used. They both do the same job prettty well. JQuery would be my personal choice. Using both frameworks means downloading cca 500 kB of the code.

Tip 6 - Each HTTP request downloads 1,2 - 2,8 MB data

According to my 10 years experience with web development, well optimized web page should load something like 150 - 300 kB, out of which 50-70% is cached.

It is OK to download even 2 MB data at first arrival on the page - if most data will be cached. But it is not okay to download about 1.5 - 2.5 MB of data with each request - just like I can see in TBG 3.2. This causes notably slow responses.

Tip 7 - Profiling result - bottleneck Geshi

It looks like Geshi takes in average more than 50% of page generation time.

Other found issues or suggestions

A. Incorrectly calculated URL webroot links for graphics

B. Inconsistent measuring of execution time

Request takes 5.81 seconds, but log window show message "Done" at 2.38 secs. What is happening between 5.81 and 2.38 secs?

C. Do not Capitalize entered username when creating new user

It is unexpected behaviour from user's point of view. It may also produce frustration if you try to capitalize accented characters. If you want to capitalize, than explicitly notify user NOT to use accented characters and use preg_match filter for that. Allowing to store accented character (LATIN2) in database may cause that database will not be able to seek correctly lowercased and uppercased characters with incorrectly set collation.

D. Redundant step: report issue

When reporting issue from dashoard, there are duplicate steps:

from menu I choose issue type
after that I must choose issue type from 3 options
on What's the issue page I still can choose issue type

At least step 2 should be skipped.

E. Minus point for redesigning creation of new users

In TBG 3.1 creating new user was on the same page. IN 3.2 creating new user will redirect to separate page. One line for whole empty screen. Without button "Add new user".

F. PHP Casper framework - where can I find documentation?

It is really a pity, that TBG has not been built on the top of some well tested framework. I am not judging whether Casper is good or bad framework... I just could not find anything about it on the internet. Unlike Yii, Nette or CodeIgniter ...

Z. Recommendations

I would recommend decreasing minimum required PHP version to 5.2 in order to address broader PHP community.
Find another workaround for PCRE bug that caused increasing minimum required PCRE version 8+. Unfortunatelly all Red Hat distributions (that means also CentOS) come with a bit outdated PCRE library 7 which makes uninstallable for large part of community. Pretty big headache for heap of people caused by a minor bug, huh?

Thanx a lot for nice issue tracker.
Oh yes, any comments are welcome!

Writing database portable PHP applications »

Comments...

zegenie AT gmail DOT com

07.11.2011 09:19

# 1

Hey.

Thanks for a nice and thorough analysis. We agree with many of your points - and most of the points we agree with have been addressed as a direct result of this article, and is available in 3.2 beta 2.

However, several of the points listed above seems to be configuration issues, and some of the statistics cannot be reproduced, making them a bit inaccurate. As an example, in the network traffic analysis, your statistics suggests that you've downloaded a whopping 1MB of data for the main issue view page. Our own tests across several browsers show nothing in that area - usually well within 120 - 150k. Another thing with that analysis is the apparent missing caching of the 500k style and js - firefox reports it as being uncached, but google chrome reports it as being cached.

In any case, the numbers presented made for a good foundation to make some much needed improvements. Both the geshi issue and the db cache issues have been fixed (the db cache issue should get backported to 3.1.x as well), making the system much, much faster than before.

On a final note: PHP 5.2 will not be supported. PHP 5.2 is unsupported by the php group, it is significantly slower than 5.3 and the feature set in php 5.3 is much better than it was in 5.2. PHP 5.3 was released late 2009 - it is in no way a new technology. We don't want to sacrifice the improvements we get from php 5.3 just to stay compatible with an old, unsupported version of php.

lubosdz

08.11.2011 00:22

# 2

Hi Zegenie,

glad to help. It's sexy bugtracker:-)
few more bugs per mail...

Lubos

Joshua Dickerson

08.11.2011 03:10

# 3

I disagree with Tip 2. Then again, since TBG uses rewrites, it might not be that bad to add it to the .htaccess. Then again, you complicate it for other web servers.

Utilize more memory based caching mechanisms for Tip 1 (allow several "levels" of caching). Compressed and serialized memcache strings are faster and smaller than including a file. If APC is enabled, it is closer, but unless apc.stat = 0, you're still checking for the file.

When loading external resources, it is better to do two things: give the admin options to load locally only or load from CDN and fallback to local and have a fallback by checking to see if the files are loaded. It is far better to use a CDN first.

I pretty much agree with everything else.

lubosdz

08.11.2011 10:49

# 4

Joshua Dickerson wrote on 08.11.2011 03:10:

I disagree with Tip 2. Then again, since TBG uses rewrites, it might not be that bad to add it to the .htaccess. Then again, you complicate it for other web servers.

I agree that compression is a good thing. Perhaps I should rename Tip 2 to "Change the way how you compress minified CSS and javascripts". The problem is, that all CSS and JS files are downloaded via PHP link that applies incorrect request cache headers. For example, I found out that my FF7 and Chrome have headers like this:

Code:

Expires:                Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma:               no-cache

Therefore I think it is not being cached but downloaded with each HTTP request thus slowing down response. I guess the whole problem would be probably solved by removing those no-cache HTTP headers (however I did not test it) - e.g.:

Code:

Expires:                Thu, 01 Jan 2020 00:00:00 GMT

Lubos

srinivasm2010 AT gmail DOT com

25.11.2011 11:51

# 5

I am facing so many issues when posting new bug report for specific project.
Its not sending email to entire project team. there is no option to setup single with below screen:

title
Description
Owner
Assinged user with dropdown
serverity
prority

Please help on this
and also getting problem with javascript exceution alerting me to continue/stop script.

lubosdz

26.11.2011 22:13

# 6

srinivasm2010@gmail.com wrote on 25.11.2011 11:51:

Hi, I am sorry - I am not a developer of BugGenie. I only helped with testing and provided few advices on improving the performance.

Please contact "zegenie AT gmail DOT com" to inform the project lead or drop a note at "http://thebuggenie.wordpress.com/2011/11/04/version-3-2-beta-2-improvements-part-2".

srinivasm2010 AT gmail DOT com

13.12.2011 13:00

# 7

Hi lubosdz,

I requested "zegenie" many times about my problem of performance of Buggenie when i post new bug report its loading altmost 1 minutes. there are so many methods are involved and Json actions. Currently i am using version 3.1.3 which is not supporting 5 users can post 50 bugs in a day.

What is this application really work in real time scenarios.

I am not sure, its giving too many issues for me. In the application used "PDO " concepts but not used Store-procedures.

I am really facing so many issues with this application. Please provide me framework architecture to tuning the application from my-end. I am also ready to support Open source team.

If possible, please send me link for latest and stable version of buggenie where i can configure and test full cycle of the application with 10 users.

thanks,
Srinivas.

lubosdz

13.12.2011 21:11

# 8

Hi srinivasm2010,

if you are concerned about the performance, you should wait for new release TBG 3.2. They are working on it constantly, even though very slowly (it should have been out already in the last week of November 2011 originally). I tested TBG 3.2 and the performance is significantly improved due to rewritten caching.

Unfortunatelly, TBG 3.2 has still many bugs, missing files/templates, crashes etc etc so it is definitely not production ready. I also used to report errors but after a while I got tired of reporting it to zegenie - who unfortunatelly is not very responsive and actually he "does not encourage reporting bugs in development phase".

You can download latest version from github here:

https://github.com/thebuggenie/thebuggenie/zipball/next

Just unzip, and run web installer. You must also manually create /core/cache/ directory which is not committed into repos.

If you want to contribute you must contact zegenie or write your suggestion at

http://thebuggenie.wordpress.com/2011/11/30/version-3-2-beta-3/

Best regards,
Lubos

srinivasm2010 AT gmail DOT com

14.12.2011 08:20

# 9

Thanks lubosdz, providing few inputs from your-end.

I am waiting for Bug genie Framework Architecture. I have 6+ years of experience in LAMP stack and DRUPAL. Currently i am working DRupal framework.

Thanks,
Srinivas.

lubosdz

14.12.2011 23:07

# 10

Hi Srinivas,

I am sorry, but I cannot provide any project architecture/design diagram, because I do not have it. We analyzed the code via debugging and profiling tool (NuSphere PHPEd).

Yet, you may want to try some UML PHP generator tools - please read more here:

http://stackoverflow.com/questions/393603/php-uml-generator

Cheers,
Lubos

srinivasm2010 AT gmail DOT com

03.01.2012 14:38

# 11

Hi Lubos,

I download latest version from Github (thebuggenie-thebuggenie-3.2beta3-0-g6acc433.zip) and configured on my local system. When I browse the application and unable to see proper look& feel of the template and <header> apply css is not refecting on Firefox 3.6.25 version.

Can you please help on this resolve HTML5 template css with Firefox and IE8.0 browser also.

Thanks,
Srinivas.

lubosdz

03.01.2012 21:28

# 12

srinivasm2010@gmail.com wrote on 03.01.2012 14:38:

Hi Srinivas,

What exactly revision of Buggenie 3.2 beta 3 did you install?
Yesterday I tested revision 216 from Github and basic template with CSS (Frontpage and Dashboard) loaded correctly. I suggest downloading most recent version (there is already revision 223 available with today's commits).

If you still would have the same problem you can sent screenshot to lubosdz AT hotmail DOT com.

Regards,
Lubos

lubos

04.01.2012 00:30

# 13

Hi Srinivas,
in case you have tested latest version and you still cannot display CSS (and/or javascripts) you may want to check your virtual host configuration or .htaccess file.
The reason is, that BugGenie uses non-standard way of downloading CSS/javascript files - being downloaded as base64 encoded 500 kB stream. This may be a problem for some servers.
This issue is quite notable with IE6-7, but in my FF3.5 it worked OK (except for caching) and displayed CSS/JS properly.
So if you play with virtual host settings (e.g. turn off deflate) you may possibly find the root of the problem.
Regards, L

srinivasm2010 AT gmail DOT com

04.01.2012 07:40

# 14

HI lubosdz,

How can i download revision 223 of Buggenie 3.2 beta 3 version. i am also sending screenshot image of template issue in IE8 and Firefox browser to your email address and its working on fine on Chrome and Safari Browser.

Please help on this.
Thanks,
srinivas.

lubos

05.01.2012 00:22

# 15

srinivasm2010@gmail.com wrote on 04.01.2012 07:40:

Hi Srinivas,

you can download latest version from following link (as stated above):
https://github.com/thebuggenie/thebuggenie/tree/next

then click on the ZIP button:

and you will get latest Buggenie from github. Then in filename you can see what revision are you downloading:

If your CSS work fine in at least one browser, then server configuration is not a problem. You may try to turn off minifying CSS file. This however, requires following steps:

1. In file ../core/templates/layout.php lines 54-58:

replace code:

Code:

<?php foreach (TBGContext::getModules() as $module): ?>
      <?php if (file_exists(THEBUGGENIE_PATH . THEBUGGENIE_PUBLIC_FOLDER_NAME . DS . 'themes' . DS . TBGSettings::getThemeName() . DS . "{$module->getName()}.css")): ?>
         <?php $tbg_response->addStylesheet("{$module->getName()}.css"); ?>
      <?php endif; ?>
<?php endforeach; ?>

with code:

Code:

<?php foreach (TBGContext::getModules() as $module): ?>
	<?php 
		$file = 'themes/' . TBGSettings::getThemeName() . "/{$module->getName()}.css";
		$path = THEBUGGENIE_PATH . THEBUGGENIE_PUBLIC_FOLDER_NAME .DS. $file;
		if (file_exists($path)): ?>
		<?php 
			$url = TBGContext::getTBGPath() . $file;
			/**
			* @var TBGResponse
			*/
			$tbg_response->addStylesheet($url, false);
		?>
	<?php endif; ?>
<?php endforeach; ?>

2. In file ../themes/oxygen/theme.php (there is only 1 line inside file)

replace code:

Code:

TBGContext::getResponse()->addStylesheet('oxygen.css');

with code:

Code:

TBGContext::getResponse()->addStylesheet(TBGContext::getTBGPath() . 'themes/oxygen/oxygen.css', false);

So much for CSS.
However, most likely you also need to turn off minifying of javascript files - do like this:

3. In file ../core/lib/common.inc.php lines 340 - 350:

replace code:

Code:

$tbg_response = TBGContext::getResponse();
$tbg_response->addJavascript('prototype.js', true, true);
$tbg_response->addJavascript('jquery-1.6.2.min.js', true, true);
$tbg_response->addJavascript('builder.js');
$tbg_response->addJavascript('effects.js');
$tbg_response->addJavascript('dragdrop.js');
$tbg_response->addJavascript('controls.js');
$tbg_response->addJavascript('jquery.markitup.js');
$tbg_response->addJavascript('thebuggenie.js');
$tbg_response->addJavascript('tablekit.js');

with code

Code:

$url = TBGContext::getTBGPath() . 'js/';
$tbg_response = TBGContext::getResponse();
$tbg_response->addJavascript($url . 'prototype.js', false, true);
$tbg_response->addJavascript($url . 'jquery-1.6.2.min.js', false, true);
$tbg_response->addJavascript($url . 'builder.js', false);
$tbg_response->addJavascript($url . 'effects.js', false);
$tbg_response->addJavascript($url . 'dragdrop.js', false);
$tbg_response->addJavascript($url . 'controls.js', false);
$tbg_response->addJavascript($url . 'jquery.markitup.js', false);
$tbg_response->addJavascript($url . 'thebuggenie.js', false);
$tbg_response->addJavascript($url . 'tablekit.js', false);

Now refresh the page cache with CTRL+R and with firebug check that all JS, CSS files are loaded.

Cheers,
Lubos

srinivasm2010 AT gmail DOT com

05.01.2012 12:22

# 16

Thanks Lubosdz for providing detailed information code to uncompress css and javascript.
I have tried above instructions but same problem with Firefox 3.6 version and remaining browsers working fine. same way i getting which attached image yesteraday to your email address.

And also i noticed one bug in latest build which you have provided link once setup on my local system.

I logged as Administrator and user& secutiry tab given permission for anaymous access to "valid user account to access any content" and save the settings. After that i open same index page url on other browser and firing below error:
Fatal error: Call to a member function hasPageAccess() on a non-object in C:\apacheroot\bugtracker12\modules\main\classes\actions.class.php on line 236

Similary, I have checked disabed Openid interface from admin setting. When i saw other browser of login interface its not disable OpenId in Login interface.

Please help on above issues.

Thanks,
Srinivas.

lubos

05.01.2012 13:04

# 17

srinivasm2010@gmail.com wrote on 05.01.2012 12:22:

Hi srinivas, as I said before, I am not the developer of buggenie. buggenie you are testing from github is not stable. i know that there are missing files and fatal errors, some of them have been reported to main developer Zegenie. He is the only one who can commit fixed to github, so there is nothing I can do about it.

Please write to zegenie at zegenie AT gmail DOT com if you have other questions or wait for stable release TBG 3.2.

Cheers,
Lubos

Your name or email *	Email will be converted into something like [michael AT gmail DOT com]
Note: Offensive and unrelated comments will be deleted. Your comments *
Enter verification code *	Please enter result from the picture above.

TBG - The BugGenie performance analyze

Tip 1 - Optimize caching the table schemas

Solution:

Tip 2 - Do not gzip contents streamed down to browser (CSS, JS)

Solution:

Tip 3 - Never rely upon external resources

Tip 4 - Add indexes to the database joined columns

Tip 5 - Any reason for using JQuery and Prototype together?

Tip 6 - Each HTTP request downloads 1,2 - 2,8 MB data

Tip 7 - Profiling result - bottleneck Geshi

Other found issues or suggestions

A. Incorrectly calculated URL webroot links for graphics

B. Inconsistent measuring of execution time

C. Do not Capitalize entered username when creating new user

D. Redundant step: report issue

E. Minus point for redesigning creation of new users

F. PHP Casper framework - where can I find documentation?

Z. Recommendations

Thanx a lot for nice issue tracker. Oh yes, any comments are welcome!

Thanx a lot for nice issue tracker.
Oh yes, any comments are welcome!