Mike Travis 41df0d61c2 x86: Add performance variants of cpumask operators
* Increase performance for systems with large count NR_CPUS by limiting
    the range of the cpumask operators that loop over the bits in a cpumask_t
    variable.  This removes a large amount of wasted cpu cycles.

  * Add performance variants of the cpumask operators:

    int cpus_weight_nr(mask)	     Same using nr_cpu_ids instead of NR_CPUS
    int first_cpu_nr(mask)	     Number lowest set bit, or nr_cpu_ids
    int next_cpu_nr(cpu, mask)	     Next cpu past 'cpu', or nr_cpu_ids
    for_each_cpu_mask_nr(cpu, mask)  for-loop cpu over mask using nr_cpu_ids

  * Modify following to use performance variants:

    #define num_online_cpus()	cpus_weight_nr(cpu_online_map)
    #define num_possible_cpus()	cpus_weight_nr(cpu_possible_map)
    #define num_present_cpus()	cpus_weight_nr(cpu_present_map)

    #define for_each_possible_cpu(cpu) for_each_cpu_mask_nr((cpu), ...)
    #define for_each_online_cpu(cpu)   for_each_cpu_mask_nr((cpu), ...)
    #define for_each_present_cpu(cpu)  for_each_cpu_mask_nr((cpu), ...)

  * Comment added to include/linux/cpumask.h:

    Note: The alternate operations with the suffix "_nr" are used
	  to limit the range of the loop to nr_cpu_ids instead of
	  NR_CPUS when NR_CPUS > 64 for performance reasons.
	  If NR_CPUS is <= 64 then most assembler bitmask
	  operators execute faster with a constant range, so
	  the operator will continue to use NR_CPUS.

	  Another consideration is that nr_cpu_ids is initialized
	  to NR_CPUS and isn't lowered until the possible cpus are
	  discovered (including any disabled cpus).  So early uses
	  will span the entire range of NR_CPUS.

    (The net effect is that for systems with 64 or less CPU's there are no
     functional changes.)

For inclusion into sched-devel/latest tree.

Based on:
	git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    +   sched-devel/latest  .../mingo/linux-2.6-sched-devel.git

Cc: Paul Jackson <pj@sgi.com>
Cc: Christoph Lameter <clameter@sgi.com>
Reviewed-by: Paul Jackson <pj@sgi.com>
Reviewed-by: Christoph Lameter <clameter@sgi.com>
Signed-off-by: Mike Travis <travis@sgi.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-05-23 18:23:38 +02:00
..
2008-04-10 15:34:05 -07:00
2008-02-03 17:48:52 +02:00
2005-10-18 08:26:15 -07:00
2007-05-11 05:38:25 -04:00
2007-07-16 09:05:50 -07:00
2007-07-17 10:23:04 -07:00
2006-06-25 10:01:20 -07:00
2008-02-03 17:48:52 +02:00
2005-04-16 15:20:36 -07:00
2006-06-25 10:01:20 -07:00
2007-05-10 18:24:13 +02:00
2005-04-16 15:20:36 -07:00
2008-05-01 08:03:58 -07:00
2005-04-16 15:20:36 -07:00
2008-04-29 08:11:16 -07:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2008-05-14 19:11:14 -07:00
2007-10-19 11:53:41 -07:00
2008-05-01 08:04:00 -07:00
2006-06-20 20:24:58 -07:00
2006-10-06 08:53:40 -07:00
2008-01-24 20:40:05 -08:00
2008-05-18 23:35:43 -05:00
2007-10-19 11:53:41 -07:00
2005-04-16 15:20:36 -07:00
2008-04-28 08:58:17 -07:00
2008-01-30 13:33:00 +01:00
2007-10-17 08:42:52 -07:00