C/C++ tip: How to measure elapsed real time for benchmarking

Topics: C/C++

API functions to get the real time (wall-clock time) at sub-second resolution differ between Windows, Linux, OSX, BSD, Solaris, and other UNIX-style OSes. This article provides a cross-platform function to get the real time, and explains what works on what OS.

How to get the elapsed real time

For benchmarking, we're interested in the elapsed real world time between the start and end of some code of interest. For this purpose, we need time values with sub-second accuracy and we don't care about time zones and date formats. This eliminates most of the API calls available in today's OSes because they only return times in seconds and produce nicely formatted strings for display to a human.

What we need is a low-level notion of time that reliably ticks over every few nanoseconds. We'll query that time before and after the code we're benchmarking, compute the difference and report that as the elapsed time. While all OSes provide the necessary API calls, each OS has its quirks.

Code

The following getRealTime( ) function works for most OSes (copy and paste, or download getRealTime.c). On OSes that need it, link with librt to get POSIX timers (e.g. AIX, BSD, Cygwin, HP-UX, Linux, and Solaris, but not OSX). Otherwise, the default libraries are sufficient.

See the sections that follow for discussion, caveats, and why this code requires so many #ifdef's.

/*
 * Author:  David Robert Nadeau
 * Site:    http://NadeauSoftware.com/
 * License: Creative Commons Attribution 3.0 Unported License
 *          http://creativecommons.org/licenses/by/3.0/deed.en_US
 */
#if defined(_WIN32)
#include <Windows.h>

#elif defined(__unix__) || defined(__unix) || defined(unix) || (defined(__APPLE__) && defined(__MACH__))
#include <unistd.h>	/* POSIX flags */
#include <time.h>	/* clock_gettime(), time() */
#include <sys/time.h>	/* gethrtime(), gettimeofday() */

#if defined(__MACH__) && defined(__APPLE__)
#include <mach/mach.h>
#include <mach/mach_time.h>
#endif

#else
#error "Unable to define getRealTime( ) for an unknown OS."
#endif





/**
 * Returns the real time, in seconds, or -1.0 if an error occurred.
 *
 * Time is measured since an arbitrary and OS-dependent start time.
 * The returned real time is only useful for computing an elapsed time
 * between two calls to this function.
 */
double getRealTime( )
{
#if defined(_WIN32)
	FILETIME tm;
	ULONGLONG t;
#if defined(NTDDI_WIN8) && NTDDI_VERSION >= NTDDI_WIN8
	/* Windows 8, Windows Server 2012 and later. ---------------- */
	GetSystemTimePreciseAsFileTime( &tm );
#else
	/* Windows 2000 and later. ---------------------------------- */
	GetSystemTimeAsFileTime( &tm );
#endif
	t = ((ULONGLONG)tm.dwHighDateTime << 32) | (ULONGLONG)tm.dwLowDateTime;
	return (double)t / 10000000.0;

#elif (defined(__hpux) || defined(hpux)) || ((defined(__sun__) || defined(__sun) || defined(sun)) && (defined(__SVR4) || defined(__svr4__)))
	/* HP-UX, Solaris. ------------------------------------------ */
	return (double)gethrtime( ) / 1000000000.0;

#elif defined(__MACH__) && defined(__APPLE__)
	/* OSX. ----------------------------------------------------- */
	static double timeConvert = 0.0;
	if ( timeConvert == 0.0 )
	{
		mach_timebase_info_data_t timeBase;
		(void)mach_timebase_info( &timeBase );
		timeConvert = (double)timeBase.numer /
			(double)timeBase.denom /
			1000000000.0;
	}
	return (double)mach_absolute_time( ) * timeConvert;

#elif defined(_POSIX_VERSION)
	/* POSIX. --------------------------------------------------- */
#if defined(_POSIX_TIMERS) && (_POSIX_TIMERS > 0)
	{
		struct timespec ts;
#if defined(CLOCK_MONOTONIC_PRECISE)
		/* BSD. --------------------------------------------- */
		const clockid_t id = CLOCK_MONOTONIC_PRECISE;
#elif defined(CLOCK_MONOTONIC_RAW)
		/* Linux. ------------------------------------------- */
		const clockid_t id = CLOCK_MONOTONIC_RAW;
#elif defined(CLOCK_HIGHRES)
		/* Solaris. ----------------------------------------- */
		const clockid_t id = CLOCK_HIGHRES;
#elif defined(CLOCK_MONOTONIC)
		/* AIX, BSD, Linux, POSIX, Solaris. ----------------- */
		const clockid_t id = CLOCK_MONOTONIC;
#elif defined(CLOCK_REALTIME)
		/* AIX, BSD, HP-UX, Linux, POSIX. ------------------- */
		const clockid_t id = CLOCK_REALTIME;
#else
		const clockid_t id = (clockid_t)-1;	/* Unknown. */
#endif /* CLOCK_* */
		if ( id != (clockid_t)-1 && clock_gettime( id, &ts ) != -1 )
			return (double)ts.tv_sec +
				(double)ts.tv_nsec / 1000000000.0;
		/* Fall thru. */
	}
#endif /* _POSIX_TIMERS */

	/* AIX, BSD, Cygwin, HP-UX, Linux, OSX, POSIX, Solaris. ----- */
	struct timeval tm;
	gettimeofday( &tm, NULL );
	return (double)tm.tv_sec + (double)tm.tv_usec / 1000000.0;
#else
	return -1.0;		/* Failed. */
#endif
}

Usage

To benchmark an algorithm's real time use, call getRealTime( ) at the beginning and end, then report the difference. It is not safe to assume the value returned by one function call has any meaning.

double startTime, endTime;

startTime = getRealTime( );
...
endTime = getRealTime( );

fprintf( stderr, "Real time used = %lf\n", (endTime - startTime) );

Discussion

Each OS has one or more ways of returning the current value of a real time clock. However, some are faster or have better resolution than others.

OS clock_gettime gethrtime GetSystemTimeAsFileTime and
Get SystemTimePreciseAsFileTime
gettimeofday mach_absolute_time time
AIX yes     yes   yes
BSD yes     yes   yes
HP-UX yes yes   yes   yes
Linux yes     yes   yes
OSX       yes yes yes
Solaris yes yes   yes   yes
Windows     yes      

Each of these is discussed below.

GetSystemTimeAsFileTime( ) and GetsystemTimePreciseAsFileTime( )

On Windows and Cygwin (Linux compatibility tools under Windows), GetSystemTimeAsFileTime( ) and GetSystemTimePreciseAsFileTime( ) both fill a FILETIME struct with the current system time. The functions are equivalent, but GetSystemTimePreciseAsFileTime( ) aims at providing a higher precision value, though it's only available in Windows 8 or Windows Server 2012 or later.

The FILETIME struct contains these fields:

typedef struct _FILETIME
{
  DWORD dwLowDateTime;
  DWORD dwHighDateTime;
} FILETIME, *PFILETIME;

The struct's fields provide the low and high 32-bits of a 64-bit time in units of 100 nanoseconds (1/10000000-th of a second).

Availability of GetSystemTimeAsFileTime( ): Cygwin and Windows 2000 and later.

Availability of GetSystemTimePreciseAsFileTime( ): Windows 8 and Windows Server 2012 and later.

Get real time:

#include <Windows.h>
...

	FILETIME tm;
	ULONGLONG t;
#if defined(NTDDI_WIN8) && NTDDI_VERSION >= NTDDI_WIN8
	/* Windows 8, Windows Server 2012 and later. ---------------- */
	GetSystemTimePreciseAsFileTime( &tm );
#else
	/* Windows 2000 and later. ---------------------------------- */
	GetSystemTimeAsFileTime( &tm );
#endif
	t = ((ULONGLONG)tm.dwHighDateTime << 32) | (ULONGLONG)tm.dwLowDateTime;
	return (double)t / 10000000.0;

clock_gettme( )

On most POSIX-compliant OSes, clock_gettime( ) (see manual pages for AIX, BSD, HP-UX, Linux, and Solaris) provides the most accurate system time. The function's first argument selects a "clock id" and the second is a timespec struct filled in with a time in seconds and nanoseconds. For most OSes, the application must link with librt.

While the timespec struct can hold a time value down to 1 nanosecond, few systems (if any) provide times at that resolution. HP-UX documentation, for instance, notes that clock_gettime( ) just calls gettimeofday( ) (see below), which has microsecond resolution. But the HP-UX kernel clock only ticks over once every 1/100th of a second, which limits gettimeofday( ) and clock_gettime( ) to 1/100th of a second resolution. Other OSes have similar quirks.

There are a few caveats that make clock_gettime( ) difficult for cross-platform code:

  • The function is an optional part of the POSIX specification and is only available if _POSIX_TIMERS is defined in <unistd.h> with a value greater than 0. Currently, AIX, BSD, HP-UX, Linux, and Solaris support the function, but OSX does not.
  • The clock_getres( ) function returns the clock resolution. But this function isn't always fully implemented. Currently it works on AIX, BSD, HP-UX, and Linux, but fails on Solaris.
  • The POSIX specification defines the names of several standard "clock id" values, including CLOCK_MONOTONIC and CLOCK_REALTIME to get the system time (more on the differences between the two below). However, currently HP-UX omits CLOCK_MONOTONIC and Solaris defines it within include files, but not within its documentation. Solaris adds its own non-standard CLOCK_HIGHRES clock, Linux adds CLOCK_MONOTONIC_RAW, and BSD adds CLOCK_MONOTONIC_PRECISE (and several other variants).

The clock IDs available on supported OSes include:

OS Available clocks
AIX CLOCK_MONOTONIC, CLOCK_REALTIME
BSD CLOCK_MONOTONIC, CLOCK_MONOTONIC_PRECISE, CLOCK_REALTIME
HP-UX CLOCK_REALTIME
Linux CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW, CLOCK_REALTIME
Solaris CLOCK_MONOTONIC, CLOCK_REALTIME, CLOCK_HIGHRES

These caveats mean a chain of #ifdef checks are required, and the ability to fall through to one of the other real time functions when they fail.

The different clocks are:

  • CLOCK_REALTIME (AIX, BSD, HP-UX, Linux, and Solaris) reports a settable (by root) time that can be mapped to the current real-world date and time. The value is the elapsed time since "the Epoch", which is usually Jan 1st, 1970 at midnight GMT (this may vary among OSes and should not be counted on). CLOCK_REALTIME is the system time used for human time display, so it's subject to corrections whenever the system checks a time server or the system administrator updates the clock. If such adjustments are made during benchmarking, then the elapsed time between two queries of CLOCK_REALTIME will be wrong. One of the other clocks should be used instead, if available.
  • CLOCK_MONOTONIC (AIX, BSD, Linux, and Solaris) reports an unsettable time that counts upwards starting at an unspecified time zero in the past that's typically the system's boot time. The time from CLOCK_MONOTONIC cannot be mapped to the current real-world date and time, but it's the right tool for benchmarking where we only need to find the elapsed time of an algorithm.
  • CLOCK_MONOTONIC_PRECISE (BSD) is a variant of CLOCK_MONOTONIC that aims at providing a more precise time.
  • CLOCK_MONOTONIC_RAW (Linux 2.6.28 or later) is a variant of CLOCK_MONOTONIC that aims for sa more precise time by accessing the underlying "raw" hardware-based time.
  • CLOCK_HIGHRES (Solaris) is equivalent to CLOCK_MONOTONIC and aims at returning a high-resolution time. The returned value is also available from gethrtime( ) (see below).

Availability of clock_gettime( ): AIX, BSD, Cygwin, HP-UX, Linux, and Solaris.

Availability of clock_getres( ): AIX, BSD, Cygwin, HP-UX, and Linux, but it fails on Solaris.

Get real time:

#include <time.h>
...

#if defined(_POSIX_TIMERS) && (_POSIX_TIMERS > 0)
	struct timespec ts;
#if defined(CLOCK_MONOTONIC_PRECISE)
	/* BSD. --------------------------------------------- */
	const clockid_t id = CLOCK_MONOTONIC_PRECISE;
#elif defined(CLOCK_MONOTONIC_RAW)
	/* Linux. ------------------------------------------- */
	const clockid_t id = CLOCK_MONOTONIC_RAW;
#elif defined(CLOCK_HIGHRES)
	/* Solaris. ----------------------------------------- */
	const clockid_t id = CLOCK_HIGHRES;
#elif defined(CLOCK_MONOTONIC)
	/* AIX, BSD, Linux, POSIX, Solaris. ----------------- */
	const clockid_t id = CLOCK_MONOTONIC;
#elif defined(CLOCK_REALTIME)
	/* AIX, BSD, HP-UX, Linux, POSIX. ------------------- */
	const clockid_t id = CLOCK_REALTIME;
#else
	const clockid_t id = (clockid_t)-1;	/* Unknown. */
#endif
	if ( id != (clockid_t)-1 && clock_gettime( id, &ts ) != -1 )
		return (double)ts.tv_sec +
			(double)ts.tv_nsec / 1000000000.0;
#endif

gethrtime( )

On HP-UX and Solaris, gethrtime( ) (see manual pages for HP-UX and Solaris) returns a high-resolution time measured in nanoseconds. On Solaris, the value is equivalent to the value returned by clock_gettime( ) with CLOCK_HIGHRES. Like that clock, the time reported by gethrtime( ) is measured from an unspecified starting point and is not subject to time server and administrator changes, which makes it good for benchmarking.

Availability of gethrtime( ): HP-UX, Solaris.

Get real time:

#include <sys/time.h>
...

	return (double)gethrtime( ) / 1000000000.0;

gettimeofday( )

On POSIX OSes, gettimeofday( ) (see manual pages for AIX, BSD, HP-UX, Linux, OSX, and Solaris) returns the current system time in a timeval struct with the time in seconds and microseconds:

struct timeval
{
	time_t      tv_sec;
	suseconds_t tv_usec;
};

Time values are real-world time measured since "the Epoch", like CLOCK_REALTIME with clock_gettime( ) above. The time returned is subject to adjustments from time server updates or the administrator setting the clock, so benchmarking should only use this time as a fallback when none of the monotonic times with clock_gettime( ) are available.

The first argument to the function is a pointer to a timeval struct to fill. The second argument optionally points to a legacy timezone struct that has been deprecated due to inadequacies in its design. Passing a NULL for the second argument is common and skips the timezone information, which isn't needed for benchmarking anyway.

Availability of gettimeofday( ): AIX, BSD, Cygwin, HP-UX, Linux, OSX, and Solaris.

Get real time:

#include <sys/time.h>
...

	struct timeval tm;
	gettimeofday( &tm, NULL );
	return (double)tm.tv_sec + (double)tm.tv_usec / 1000000.0;

mach_absolute_time( )

On OSX, mach_absolute_time( ) returns a time measured in multiples of a time base returned by mach_timebase_info( ). The later function fills a mach_timebase_info struct with a numerator and denominator:

struct mach_timebase_info
{
	uint32_t numer;
	uint32_t denom;
};

To convert the value returned by mach_absolute_time( ) to seconds, multiply it by the time base numerator, divide by the denominator, then divide by 1,000,000,000.0 to convert from nanoseconds to seconds. Since the time base doesn't change, the struct can be retrieved once and re-used each time mach_absolute_time( ) is called.

On OSX, mach_absolute_time( ) may be more accurate than gettimeofday( ) and is recommended by Apple. The time is monotonic, like CLOCK_MONOTONIC for clock_gettime( ) above, so it cannot be converted to a real-world date and time, isn't settable, and doesn't change on time server updates or administrator changes.

Availability of mach_absolute_time( ): OSX

Get real time:

#include <mach/mach.h>
#include <mach/mach_time.h>
...

	static double timeConvert = 0.0;
	if ( timeConvert == 0.0 )
	{
		mach_timebase_info_data_t timeBase;
		(void)mach_timebase_info( &timeBase );
		timeConvert = (double)timeBase.numer /
			(double)timeBase.denom /
			1000000000.0;
	}
	return (double)mach_absolute_time( ) * timeConvert;

time( )

On POSIX OSes, time( ) (see manual pages for AIX, BSD, HP-UX, Linux, OSX, and Solaris) returns the current system time in seconds. The value is measured since "the Epoch" like gettimeofday( ) and the CLOCK_REALTIME clock for clock_gettime( ) above. Like them, the system time can jump forward or back from time server updates or changes by the administrator, which makes it poor for use in benchmarking.

The value returned by time( ) is in seconds while the earlier gettimeofday( ) returns time in microseconds. Since POSIX defines both functions, all POSIX-compliant OSes have both and there is never a good reason to use the lower-resolution time( ) instead of gettimeofday( ).

Availability of time( ): AIX, BSD, Cygwin, HP-UX, Linux, OSX, and Solaris.

Get real time:

#include <time.h>
...

	return (double)time( NULL );

Other approaches

Some processors (Intel x86, Itanium, and others) include a Time Stamp Counter register that contains a 64-bit number that increments on every clock tick. The register is available from user code with a bit of assembly. While the register has been useful for benchmarking in the past, modern multi-processor systems and out-of-order execution make this an undependable method today.

On HP-UX, "Project Mercury" introduces memory pages shared between an application and the kernel in order to speed up some system calls. The hg_gethrtime( ) function is the Mercury version of gethrtime( ). Mercury is only available as of HP-UX 11.31 and requires #include <mercury.h> and linking with libhg.

On AIX, the gettimer( ) function fills a timestruct_t or itimerspec struct with the time in seconds and nanoseconds. The function's first argument selects a well-known timer, but the only one currently defined is TIMEOFDAY. This is the equivalent of the CLOCK_REALTIME clock supported by clock_gettime( ) (see above).

On POSIX systems, the obsolete ftime( ) function fills a timeb struct with the time in seconds and milliseconds. In 2008 the function was removed from the POSIX specification and its further use is strongly discouraged. The function remains available on AIX (with the libbsd compatibility library), BSD, HP-UX, Linux, OSX, and Solaris.

Downloads

Further reading

Related articles at NadeauSoftware.com

Web articles

Comments

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

Nadeau software consulting
Nadeau software consulting