Benchmark toolbar

Notice: the details Bugfoo's use of memory and execution time, and about its relative accuracy, were taken from the non-encoded version. Since it has been encoded, no major differences have been noted in memory use and execution time in the few tests we have been able to carry out, and we will provide more accurate information as it becomes available.
The tool xp memory usage has not yet been included in a test version; it will be available later, probably also as a separate download for those who already have a test version of Bugfoo.

The Benchmark toolbar contains a number of tools which can be useful for benchmarking-type tasks -- that is, determining how fast code executes, and what resources it needs, consumes, or shares. Because these tools rely on the operating system for the information they provide, they are operating-system specific and not all are available on all operating systems. Windows users in general will be able to use adjusted microtime and timer, and Windows XP users will also be able to use xp memory usage. Linux users will be able to use adjusted microtime, timer, memory usage, resource usage, load average, and cpu load; Mac OSX users will be able to use adjusted microtime, timer, memory usage, and resource usage.

Of the benchmark tools, the tool adjusted microtime, memory usage, and resource usage are probably the most useful; load average and cpu load are also provided, but you will probably use the first three more frequently.

adjusted microtime

The tool adjusted microtime tells you how long script execution has taken in seconds and microseconds, and also gives you the additional information of the script's execution time excluding the execution time used by Bugfoo. Bugfoo keeps track of the time that it spends in processing information, so this second value is simply the difference between the total execution time and Bugfoo's execution time.

The tool also provides a figure called "relevance." Relevance is simply a ratio of the time spent in the script iteself, and not Bugfoo, since the last time adjusted microtime was called, compared to the total execution time since the last time it was called -- normally on a scale from 0 to 10. Suppose you have this tool set, and no other Bugfoo tools are set, and a report point is called at the beginning of your script, and then again in the middle of script execution, after a database query has been made. This figure will be very high, maybe even 9.9. This is because Bugfoo has had to do very little work -- only this very lightweight tool -- which, in the example in the screenshot below, is only about 0.003 seconds (if you're curious, this was done on a P4 2.4MHz computer running Ubuntu Linux with a full set of development servers and other resource-hogging development software, so it runs rather slowly). However, suppose you activate a bunch of tools for report points in Bugfoo: you have Bugfoo show you the highlighted code context with show code lines, you are searching through all the included files with a regular expression, and you are dumping a few very, very big arrays. If report points are called fairly close to one another, the relevance factor will be close to zero. This is because between these two report points Bugfoo had a whole lot of work to do, while not very much happened in the script between the two report points.

Another thing to keep in mind concerning the relevance factor is the fact that a low relevance factor probably means that even the corrected execution time value is likely to be influenced by Bugfoo's operation. If you first run your script with adjusted microtime alone, and then do the same page request with adjusted microtime and seven other resource-hungry tools, you might expect the total execution times to be longer, since this includes the time Bugfoo uses, but the adjusted times to be the same, since the script itself hasn't changed, and adjusted time represents the script time. This probably won't be the case. One busy, resouce-intensive part of a script usually also slows down other parts of a script since so many resources are being used, so over-using Bugfoo tools will also affect the adjusted time.

At the time of this writing, shortly before the pre-alpha test version is released, there are problems with the adjustment figure; it is not always reliable -- particularly when you are assigning Bugfoo a fairly heavy load, or when an autostart or early report point is called. It is almost certainly unreliable (and shows ridiculous figures) when the adjusted figures are larger than the unadjusted figures, which occurs occasionally when Bugfoo itself has a greater processing load than the script itself. Otherwise, it is usually fairly reliable. We hope to fix this before the alpha version is released.

Adjusted microtime is good for use with other benchmarking tools; it uses very little script execution time itself, so it will distort your benchmarking results only minimally, but it will also give you a relative idea of what kind of resources Bugfoo itself is using. This is very important in benchmarking, since what you want to know about is how the script itself is performing, not the application which is measuring the script. For more information on benchmarking in general, read the section on benchmarking in general below.

timer

The timer tool is for measuring the execution time of a specific fragment of code, compared to the total execution time. Calling it from an error point or a report point will, the first time it is called, start a virtual stopwatch; the next time the tool is called, the stopwatch is stopped, and the total stopwatch time is displayed, along with the percentage of total execution time this represents -- the third time it is called again, the stopwatch is started again, etc ... while a timer is also started which measures the total time of execution since the first time the tool was called. Each time the point is called, whether starting or stopping, it also shows standard adjusted microtime information (see above). This tool is the most powerful when you call it both as a fullpage tool, and as a report point or error tool -- when it is called as a fullpage tool, on the beginning of execution, it starts the total execution time timer (but does not start the stopwatch), and each time a report point or error point is called, in addition to starting or stopping the stopwatch, it also saves the stopwatch's time information. When the fullpage calls the tool again after execution, it displays the total stopwatch time, plus the relative percentage time of execution for each time the stopwatch was started and stopped. The number in the left column is the number of the error point or report point where the stopwatch "stopped" and recorded the information -- there are rows for each time the timer was started and stopped, showing the total time, the percent of the total execution time that this time accounts for, and the percentage of the total timer time (stopwatch total) that this accounts for -- and then these same three figures again, but in "adjusted time" (which reflects the script-time only, without Bugfoo's own execution time).

For example, if your script consistently makes database queries from one part of the script, you can add triggered errors directly before and after the call to mysql_query, and call this tool, along with local variable and the name of the variable holding the query filled in the name field of the php etc toolbar. This will allow you to see the percentage of execution time each of your individual queries is making.

To use this tool well you really have to make sure that the timer will always be started and stoped at the appropriate times. For example, if you want to measure the amount of time a particular function takes each time it is called, put a triggered error (or report point) at the beginning of the function, and also put one right before each return statement, as well as at the end of the function. This way the timer will always be turned off when the function has finished and returns.

memory usage

This tool shows the amount of memory which is currently allocated to the script. It is a useful indicator of how much memory your script is using since, as your script actually uses more memory, more memory is allocated to it. Memory is also de-allocated from your script when parts of memory (e.g. variables) aren't needed any more.

memory usage also shows you the difference in memory usage since the last time it was called. Ouput is very simple -- first the number of bytes, then the same number in terms of kilobytes. The difference of memory usage from the last time this tool was called is then shown in parentheses, with plus or minus marks.

Keep in mind that Bugfoo also uses memory, which is included in this figure. If you want these results to be as representative as possible of your script's memory usage, turn off all other tools, or at least any tools which are process-intensive or which produce a great deal of output. Also, don't call report points too often -- call them only at the points where you want memory usage to be measured.

When verbose mode is on, memory usage also provides you with an adjusted figure, which adjusts the amount of memory used by subtracting the current size of output which Bugfoo has produced plus a number corresponding to the basic, minimal use of memory by Bugfoo. (screenshot).

How accurate is the adjusted memory figure that this tool gives? It is much more accurate when the tool is used alone, with no other tools activated. We haven't been able to test this thoroughly yet, but one short test showed the tool's adjusted memory usage at about 98% accuracy when 129 report points were called (which is more than you will usually want), and another showed adjusted memory usage at better than 99% accuracy when 90 report points were called (test was simply to insert a few report points in a class that was called repeatedly during the script, then remove all report points, and compare the adjusted memory usage figure). This seems to indicate that the adjusted memory is highly accurate when less than 100 report points are called, and loses accuracy as the number of report points rises. However, doing the same test with 129 report points shows an accuracy of only 96% when the extra tools html size and included files are activated. And when report points tools included files, show code lines, and find in variables were activated with a multi-line regex searchstring, along with included files activated for the fullpage, accuracy of adjusted memory dropped to a very unacceptable 80%. So use this tool wisely, by turning off other tools, if you want accurate results. Please note that these indications are based only on a single trial, so they are mere indicators and not confirmed data -- more tests will be held when we have more time.

This tool is only available when the PHP tool get_memory_usage() is available (PHP 4.3.2 +), and it probably won't be available on Windows platforms. If it is available, it can be used with the fullpage, report points, and errors.

notes for advanced and finicky users:: If you are trying to squeeze out the most precise results possible from this tool, you should know that if you call this tool with start and finish set for report points, the "start" report point will register about 2.5K more than Bugfoo uses during the rest of execution. The "start" point shows you basically what memory is being used without the script having been loaded. If you set memory usage on the fullpage on a blank file, you will see a non-adjusted return of about -23K -- a phenomenon similar to the 2.5K more value with report points. The "adjustment" feature subtracts a fixed amount plus the length of Bugfoo output from the amount of memory reported; this fixed amount can also be modified if your system does not seem to be reflecting this same amount of memory that Bugfoo and the memory tool together use. To really fine-tune this adjustment, you can "calibrate" it by making a blank file (without any opening <?php tag - just an empty file) named a.php inside the main directory in any of the scripts you have in the bugfoo_scripts directory, and request this file using the tool memory usage set for the fullpage and the setting verbose mode on. Add the "bytes" figure (the first number displayed, without kB following it) of memory usage (the first blue number) to length Bugfoo output, and change the value of $adjustment in this tool's config.php file to this value. This "calibration" won't affect the relative accuracy of the memory usage tool (how much memory increases or decreases), and only the "absolute" value, and then probably only by a few bytes -- so it is probably not worthwhile for most users.

notice: xp memory usage is not included in the current test version

xp memory usage

This tool is provided for Windows XP users for some limited access to information regarding script memory usage. It is limited since the tool relies on making a system call to the windows tasklist, which very quickly opens and closes a windows command prompt box each time the tool is called. For this reason, the tool is limited (by default) to being called six times, after which it no longer makes the system call or provides memory information, in order to prevent instability. It uses the same method as described in the user-contributed notes in the PHP manual page on the function memory_get_usage(). It is not an ideal way of getting information about memory usage, but at the moment it is the only way we know of for providing you with this information if you are using Windows. You can change the maximum number of times this tool is called in the tool's config.php file.

a few diverse memory-related facts regarding Bugfoo

We thought you might be interested in Bugfoo's own usage of memory -- or at least, the amount of memory it gets allocated -- and this seemed like one of the most relevant places to put them -- so here are some facts for you. All "facts" come from a simple, single test done on a P4 2.4Mhz machine with 1GB memory on Ubuntu Linux.

• a single-line file shows a memory usage of 8.484K
• the same single-line file placed in the Bugfoo scripts directory with Bugfoo off shows a memory usage of 14.5K (this is due to the Bugfoo prepended file, but is a rather miniscule amount of memory by PHP standards)
• Bugfoo, all by itself, uses about 250K.

for comparison:

• phpBB2 uses 746.5K on its front page when there is only one forum board, with one thread containing the single article contained in the default install
• WordPress 1.5.1.3 with Headline Images installed and 11 single-line articles uses 2,844K to generate the front page
• OSCommerce uses 1,006K for a typical product-view page
• Xaraya 1.0.0 RC3 uses 4,331K in the default front page without any artices
• EZPublish uses 12,562K for the default "news" page with the two standard lorem ipsum article summaries
• curious how much memory your scripts are using?

resource usage

resource usage is a tool which allows you to see how much computing resources your script alone, in the one pageview (plus any Bugfoo tools you have enabled) is using. If your computer is slowed down at some point by a number of other processes, adjusted microtime will give you a larger, slower figure for your script's execution -- your computer is busy, so it takes longer for your script to execute. resource usage, on the other hand, checks the values user time and system time for that particular process your script is running at the moment. So the results it delivers are not as likely to be skewed as by calling adjusted microtime. "User time" is the processing time the script needs in general "by itself"; "system time" is the processing time used by the operating system for operating system operations which have been called by the script -- e.g., reading and writing files.

When either of these values have changed, that value is displayed.

If the number of swaps has changed (your computer using the hard disk to store memory that would otherwise be in RAM -- something you want to avoid -- "swaps" registering 0, or not showing up at all, is best), this information is also displayed.

Resource usage does have some degree of latency (see below) -- so if you, e.g., call two report points directly after one another, no information will be provided, since the user and system time for the script will not have been updated, showing no change.

In addition to user time and system time, if you call this tool with verbose mode on, a good deal of other memory-related information is displayed when it has changed -- depending on your system, information including voluntary and involuntary context switches, page reclaims, page faults and other information (screenshot). Like system time, each item of this information is only shown when it has changed.

This extra information in verbose mode is not likely to be very useful to you when assessing your script's performance. It varies greatly across identical requests. It is provided mainly because this information is given with the other information the tool requests, just in case you may have some possible use for it. If you have no idea what these things are, don't sweat -- they're likely to be interesting only to diehard techaholics.

Unlike adjusted microtime and memory usage, there is no "adjustment" available for this tool -- Bugfoo shares the same process as the script you are measuring, and there is no criterion for judging which portion of user time has been used by Bugfoo. However, you can run this tool together with adjusted microtime, and look at the "relevance" factor for an estimate as to what resources the tool is using. It is an extremely light tool, so it won't use considerable resources if you only use it by itself, and don't call it repeatedly in rapid succession.

This tool is not available to Windows users (sorry -- Windows doesn't support the type of calls it makes), and it might not work on Mac and BSD platforms.

load average

This tool at the moment is only available for Linux, but may later be made available for Mac and BSD systems. Sorry Windows users - your operating system doesn't make this kind of information easily accessible, but you can see similar information from your comuter by pressing control-alt-delete if you are on a more recent version of Windows.

This tool shows you the average load of the computer in the last 1, 5, and 15 minutes -- the same information you get if you run top, provided so you can get this info without having to start an extra program. This is basically, in laymanish terms, the number of things the the computer still has to do in order to get done everything which the applications that are running on it have told it to do. So this figure doesn't have to do with the performance of your script alone, but rather the server and whatever else is going on on the computer where the server is located. Numbers from 0-8 are common; for an idea of what these mean, one hosting company that hosted us said that when load average numbers started to get into 5 frequently, they knew a server was overloaded with sites or had to be upgraded. (Keep in mind these are public web servers, with hundreds or thousands of sites on them ...).

You probably won't use this tool much, and when you do, it may be primarily to get an estimate of how busy your computer is at the moment. When you are running comparitive benchmarks, ideally the 1-minute load average (the first figure) should be the same at the moment you call each script. So if you call load average and it's say, 2, but you want a fairly accurate benchmark, try closing other applications and waiting a minute or twos, and then see if the load average hasn't decreased, since it's usually easiest to hold comparitive benchmarks with the load average beginning at 0. Keep in mind that your browser also contibutes to load average, and in most cases, your browser will contribute more when it views a page, than PHP and your server do in generating that page. So just viewing different pages in fairly rapid succession will also raise your load average some.

In verbose mode, this tool also provides the extra information which loadavg gives you: the first is the "number of currently executing kernel entities" -- which will probably be 1 if you have a single simple processor, or more if you have multiple processors or a processor with hyperthreading -- the manpage for proc on /proc/loadavg says this will be a number less or equal to your number of CPU's, but our main development machine which has a single hyperthreading processor often shows 3, so don't go asking yourself who installed that extra CPU if this happens to you. After this number, you have a slash, followed by a number representing "the number of kernel scheduling entities that currently exist on the system", to quote the manpage. The last figure is the process identification number (PID) of the last process that has been created on the system. Again, more techie info to slake the thirst of techieholics, not necessarily useful to you if you aren't similarly inflicted.

cpu load

CPU load shows you basically how much of your CPU's resources are being utilized. In general, it's good when your CPU load is lower, because things tend to run faster -- when your CPU is 100% occupied, anything that your computer needs to get done in addition to what it's doing at the moment will simply have to wait. This figure applies to the whole system it is measured on -- and not just your script. Your browser and other applications you have running contribute to it.

However, programs which have a lot to do in one short moment are something of an exception to this rule - for a fraction of a second, they take up a large portion, up to 100%, of your CPU's resources, and then they're finished, leaving your processor completely free to take on all the other things your computer has got planned for it. PHP (like most webservers in general) is like this type of application -- it doesn't do much of anything in the "background", taking up your CPU resources by lingering around there in the background counting various odds and ends, but when you do give it work to do by requesting a page on a webserver, it can use up to 100% of your resources in a short burst of activity, sometimes in less than a tenth of a second. So you shouldn't get all concerned if running your script shows a 100% CPU usage, as long as it does not continue to do so for a number of seconds. If you are browsing your script on the same computer as where the script is running, a good portion of that CPU activity may also be due to your browser requesting the page.

The first time this tool is called, all the information it has available is the total amount of CPU time the processor has been busy, and the total amount of time is has been idle -- this total amount is the amount since you last rebooted. So the first time your call it, the CPU usage percentage reflects the percentage of CPU time occupied since the last time you rebooted, and has nothing at all to do with your script. So the tool will display something like this:

initializing: 4.78% is "absolute value" (load of your total system time since reboot)
jiffies: 0

In general, Bugfoo displays information that is "different" from standard information in grey -- this is in grey, since it has to do with your computer's load average since reboot, and not your script. It's not of immediate relevance, but you may find it handier than adjusted microtime in determining approximately "where" in terms of script runtime a particular error or report point is located, as the numbers are simple, nice, and round. If your script takes a total of 70 jiffies, and the point you are interested is at 52, you know it's about 3/4 of the way through the total runtime of your script.

The second time it is called, the relative amount (+ or - value) displayed is the difference with that value since reboot (since that was the last value).

The value jiffies basically represents time the way your operating system counts it when keeping track of the CPU usage time and other things.

on benchmarking in general

Benchmarking tools are handy for showing you how a script performs. Is it nice and quick? Is it likely to stay nice and quick, even when your site gets popular with lots of visitors hitting it at the same time? How much memory is it using? etc. Benchmarking tools can help you cut the "bloat" in your script, so it runs faster and more efficiently.

Most of the time you will use the benchmarking tools to just get a quick idea of how much execution time or memory use one part of your script uses compared to another -- or how one version of script code runs compared to another one -- informal, on-the-spot comparisons. However, at other times, you will be more concerned about the accuracy of the benchmarking information. How you use the benchmark tools and what you want to take account of will depend on the task at hand and the goals you have. Whatever it is you are using the benchmark tools for, it's helpful to be aware of what is at stake in benchmarking.

The very most scientific and accurate benchmarking results that can be achieved for an application would consist in setting up an independent mechanism for measuring that application's performance and resource usage with no effect whatsoever on the execution and performance of the script itself. This would, of course, be quite complicated, involving at least two computers at once (and in the strict, ideal sense, may even be impossible -- but that is a theoretical matter).

Bugfoo, of course, is much simpler. Since it shares the same resources as the script it measures, it isn't the ideal "uninvolved observer." It isn't a dedicated scientific benchmarking instrument, but it allows you to get script process and performance information at any point in your script with just a couple of mouseclicks. It won't produce results that are worthy of publication in Scientific American, but they might fit the bill for more popular science magazines -- and they're most likely relevant enough for articles comparing scripts in tech sites, too. But you do need to produce your information wisely. If you let Bugfoo run 7 other tools at the same time you run a benchmarking tool, your benchmarking tool won't be producing any quality information at all -- since Bugfoo itself will be using so much of your computer's resources, you won't know how your script performs in more "natural circumstances."

So, in general, when using benchmarking tools, turn off all other Bugfoo tools and options. Even the file link options consume a tiny bit of computer resources -- so you can turn them off, too. Also, keep in mind what your computer is doing at the moment. If you are comparing scripts for execution time with adjusted microtime and you test one of the scripts at the same moment a spam filter kicks in, or your computer begins an indexing process, the script which shares the computer resources with these other processes will be affected, and the execution times shown will be slower than in other circumstances. So the comparison between the scripts won't be "fair."

There are any number of things which can affect a script's performance, or cause a script's performance to vary, even for the identical page request. For example, the phpBB2 version which we sometimes test Bugfoo with -- the main index.php page, as the forum is set up of this writing, sometimes produces 13 database queries, and sometimes it produces 15 database queries. Why? We have no idea -- it may be checking some time-based information which, if older than a given number of seconds, requires refreshing with a few extra queries. Scripts which use page-caching can have enormous differences in execution times of page requests before and after the page has been cached - execution time can be more than 50 times longer when a page is requested which is not yet cached. If you use MySQL 4+, your database is also capable of caching queries, so the database queries will go faster on the second and third pageviews than on the first -- databases can also cache index keys, so the database factor alone in benchmarking becomes rather complex. The use of code cache and accellerator extensions adds yet another degree of complexity to the issue, if you have any of these installed. So in general, with any benchmarking criterion, whether it be a Bugfoo tool or a different resource for benchmarking, take your first benchmarking results with a grain of salt, and make sure to carry out your benchmarking tests a number of times if you are after anything more than an informal comparison. See if your results are consistent, and if they aren't, try to figure out what is at the root of the differences. Windows XP users can hit control-alt-delete to see the current processes which are taking place and consuming resources; Linux users can run top, or to just see the current resources being used, open gkrellm or xload. Differences can also be script-dependent, as in our phpBB2 example with the differing number of queries, if your script uses caching, or any number of other possible reasons. If you can't find what causes the differences, you must assume that if you have largely varying benchmarking results, what you are measuring are really different things, different types of script processes, and you don't have a singular benchmark result which reflects just one process. Otherwise, you can take the average of similar benchmarks as a relatively fair appraisal.

Also, don't forget that a script's greatest "bottleneck" (the thing that slows it down the most) is often -- some might even say "almost always" -- the way it interacts with a database. So when you are checking your script for performance issues, don't neglect the database tools -- they can also help you in assessing and improving your script's performance. See if you can't cut down on unnecessary queries, or improve on your queries' use of indexes, and the query cache, if you have one.

Differences in code execution efficiency are most felt in code that gets repeated over and over again -- primarily, code which is called inside loops. It's this code that you should always look at first when trying to improve the efficiency of your application. Just changing one line of code which is not located inside a loop will often give such a small difference in execution time that it isn't even noticable with a simple time test -- other background factors which affect code execution time may be larger, in which case you will hardly be able to detect any difference at all. In order to measure the efficiency of different lines of code, it is a common benchmarking practice to place them inside loops which execute the code for 1,000 times or more. Create a test directory inside the bugfoo_scripts directory for creating files just for this purpose -- you can try out different code variants inside loops there, without having to modify the original scripts' files, and change your script only after you have determined which variant of code runs most efficiently.

memory usage is likely to be more sensitive to code differences than adjusted microtime, so loops created to test memory efficiency frequently don't have to iterate so often -- you may even have significant, repeatable results without looping.

Bugfoo is a very light-weight application, and we have gone to great lengths to minimize the resources it uses. In general, if you use it with only one or two lightweight tools on, like the benchmark tools in general, it will provide highly accurate results, with minimal skewing of the results due to Bugfoo's own operation. Its strength in benchmarking is in delivering this information to you in the easiest, fastest way possible, as information which is collated and available later if you need it. However, if time is not at a premium, you will achieve more accurate results if you simply use the PHP functions microtime(), get_memory_usage() or getrusage() "hard coded" into your script at the places where you you want to measure these values. Bugfoo makes this much easier, at the cost of only a very small degree of accuracy.

Please also note that for many tasks, there are other great tools available which, for some specific benchmarking tasks, will provide accessory information -- and for some tasks, better information. Benchmarking is always best when you can compare results from a number of sources. If you have installed Xdebug, it will also provide you with very powerful tools for analyzing the performance of your scripts.

your benchmark figures and latency

"Latency" in computer terms means a period of time between an actual event, and the time necessary for information to become available on that event. All Bugfoo benchmarking tools, except for adjusted microtime, make use of information which is "polled" or checked every so often -- many times per second, but not so fast that this information is fresh with every line of code that PHP executes. This means that often, the Bugfoo benchmarking tools won't return anything, since the information has not been changed, if they care called in rapid succession. Most Bugfoo tools will simply be "skipped", or not show you anything, if no information has changed since the last time they were called. This saves you screenspace, memory, and the hassle of reading lines and lines of identical information.

Some tools use information with a higher latency than others, for example cpu load and load average.

So when tools are called in rapid succession and you don't see any information return, it is possible that these values "theoretically" should be different, but your operating system simply has not updated the information yet, and will do so in a few microseconds.

todo's for this toolbar: fix time adjustments and relevance figure for adjusted microtime