* RE: [ PATCH ] - Avoid slow TLB purges on SGI Altix systems
@ 2005-10-27 18:41 Luck, Tony
2005-10-27 18:44 ` Dean Roe
0 siblings, 1 reply; 6+ messages in thread
From: Luck, Tony @ 2005-10-27 18:41 UTC (permalink / raw)
To: Dean Roe; +Cc: linux-ia64, linux-mm
>flush_tlb_range() only calls platform_global_tlb_purge() for CONFIG_SMP,
>so there's no point in having that code in ia64_global_tlb_purge().
So you have dropped the "mm->context = 0;" for the UP case (and replaced
it with a series of ia64_ptcl() calls).
To maintain the old behaivour you need to have:
#ifndef SMP
if (mm != current->active_mm) {
mm->context = 0;
return;
}
#endif
in the start of flush_tlb_range().
-Tony
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [ PATCH ] - Avoid slow TLB purges on SGI Altix systems
2005-10-27 18:41 [ PATCH ] - Avoid slow TLB purges on SGI Altix systems Luck, Tony
@ 2005-10-27 18:44 ` Dean Roe
0 siblings, 0 replies; 6+ messages in thread
From: Dean Roe @ 2005-10-27 18:44 UTC (permalink / raw)
To: Luck, Tony; +Cc: Dean Roe, linux-ia64, linux-mm
On Thu, Oct 27, 2005 at 11:41:13AM -0700, Luck, Tony wrote:
> >flush_tlb_range() only calls platform_global_tlb_purge() for CONFIG_SMP,
> >so there's no point in having that code in ia64_global_tlb_purge().
>
> So you have dropped the "mm->context = 0;" for the UP case (and replaced
> it with a series of ia64_ptcl() calls).
>
> To maintain the old behaivour you need to have:
>
> #ifndef SMP
> if (mm != current->active_mm) {
> mm->context = 0;
> return;
> }
> #endif
>
> in the start of flush_tlb_range().
>
>
> -Tony
>
Sorry, now I see. Fixed patch coming soon.
Thanks,
Dean
--
Dean Roe
Silicon Graphics, Inc.
roe@sgi.com
(651) 683-5203
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] - Avoid slow TLB purges on SGI Altix systems
@ 2005-10-27 20:41 Dean Roe
0 siblings, 0 replies; 6+ messages in thread
From: Dean Roe @ 2005-10-27 20:41 UTC (permalink / raw)
To: tony.luck; +Cc: linux-ia64, linux-mm
flush_tlb_all() can be a scaling issue on large SGI Altix systems
since it uses the global call_lock and always executes on all cpus.
When a process enters flush_tlb_range() to purge TLBs for another
process, it is possible to avoid flush_tlb_all() and instead allow
sn2_global_tlb_purge() to purge TLBs only where necessary.
This patch modifies flush_tlb_range() so that this case can be handled
by platform TLB purge functions and updates ia64_global_tlb_purge()
accordingly. sn2_global_tlb_purge() now calculates the region register
value from the mm argument introduced with this patch.
Signed-off-by: Dean Roe <roe@sgi.com>
Index: linux-ia64-test/include/asm-ia64/machvec.h
===================================================================
--- linux-ia64-test.orig/include/asm-ia64/machvec.h
+++ linux-ia64-test/include/asm-ia64/machvec.h
@@ -26,7 +26,7 @@
typedef void ia64_mv_irq_init_t (void);
typedef void ia64_mv_send_ipi_t (int, int, int, int);
typedef void ia64_mv_timer_interrupt_t (int, void *, struct pt_regs *);
-typedef void ia64_mv_global_tlb_purge_t (unsigned long, unsigned long, unsigned long);
+typedef void ia64_mv_global_tlb_purge_t (struct mm_struct *, unsigned long, unsigned long, unsigned long);
typedef void ia64_mv_tlb_migrate_finish_t (struct mm_struct *);
typedef unsigned int ia64_mv_local_vector_to_irq (u8);
typedef char *ia64_mv_pci_get_legacy_mem_t (struct pci_bus *);
Index: linux-ia64-test/arch/ia64/mm/tlb.c
===================================================================
--- linux-ia64-test.orig/arch/ia64/mm/tlb.c
+++ linux-ia64-test/arch/ia64/mm/tlb.c
@@ -86,10 +86,15 @@
}
void
-ia64_global_tlb_purge (unsigned long start, unsigned long end, unsigned long nbits)
+ia64_global_tlb_purge (struct mm_struct *mm, unsigned long start, unsigned long end, unsigned long nbits)
{
static DEFINE_SPINLOCK(ptcg_lock);
+ if (mm != current->active_mm) {
+ flush_tlb_all();
+ return;
+ }
+
/* HW requires global serialization of ptc.ga. */
spin_lock(&ptcg_lock);
{
@@ -135,15 +140,12 @@
unsigned long size = end - start;
unsigned long nbits;
+#ifndef CONFIG_SMP
if (mm != current->active_mm) {
- /* this does happen, but perhaps it's not worth optimizing for? */
-#ifdef CONFIG_SMP
- flush_tlb_all();
-#else
mm->context = 0;
-#endif
return;
}
+#endif
nbits = ia64_fls(size + 0xfff);
while (unlikely (((1UL << nbits) & purge.mask) == 0) && (nbits < purge.max_bits))
@@ -153,7 +155,7 @@
start &= ~((1UL << nbits) - 1);
# ifdef CONFIG_SMP
- platform_global_tlb_purge(start, end, nbits);
+ platform_global_tlb_purge(mm, start, end, nbits);
# else
do {
ia64_ptcl(start, (nbits<<2));
Index: linux-ia64-test/arch/ia64/sn/kernel/sn2/sn2_smp.c
===================================================================
--- linux-ia64-test.orig/arch/ia64/sn/kernel/sn2/sn2_smp.c
+++ linux-ia64-test/arch/ia64/sn/kernel/sn2/sn2_smp.c
@@ -177,6 +177,7 @@
/**
* sn2_global_tlb_purge - globally purge translation cache of virtual address range
+ * @mm: mm_struct containing virtual address range
* @start: start of virtual address range
* @end: end of virtual address range
* @nbits: specifies number of bytes to purge per instruction (num = 1<<(nbits & 0xfc))
@@ -188,21 +189,22 @@
* - cpu_vm_mask is a bit mask that indicates which cpus have loaded the context.
* - cpu_vm_mask is converted into a nodemask of the nodes containing the
* cpus in cpu_vm_mask.
- * - if only one bit is set in cpu_vm_mask & it is the current cpu,
- * then only the local TLB needs to be flushed. This flushing can be done
- * using ptc.l. This is the common case & avoids the global spinlock.
+ * - if only one bit is set in cpu_vm_mask & it is the current cpu & the
+ * process is purging its own virtual address range, then only the
+ * local TLB needs to be flushed. This flushing can be done using
+ * ptc.l. This is the common case & avoids the global spinlock.
* - if multiple cpus have loaded the context, then flushing has to be
* done with ptc.g/MMRs under protection of the global ptc_lock.
*/
void
-sn2_global_tlb_purge(unsigned long start, unsigned long end,
- unsigned long nbits)
+sn2_global_tlb_purge(struct mm_struct *mm, unsigned long start,
+ unsigned long end, unsigned long nbits)
{
int i, opt, shub1, cnode, mynasid, cpu, lcpu = 0, nasid, flushed = 0;
+ int mymm = (mm == current->active_mm);
volatile unsigned long *ptc0, *ptc1;
- unsigned long itc, itc2, flags, data0 = 0, data1 = 0;
- struct mm_struct *mm = current->active_mm;
+ unsigned long itc, itc2, flags, data0 = 0, data1 = 0, rr_value;
short nasids[MAX_NUMNODES], nix;
nodemask_t nodes_flushed;
@@ -216,9 +218,12 @@
i++;
}
+ if (i == 0)
+ return;
+
preempt_disable();
- if (likely(i == 1 && lcpu == smp_processor_id())) {
+ if (likely(i == 1 && lcpu == smp_processor_id() && mymm)) {
do {
ia64_ptcl(start, nbits << 2);
start += (1UL << nbits);
@@ -229,7 +234,7 @@
return;
}
- if (atomic_read(&mm->mm_users) == 1) {
+ if (atomic_read(&mm->mm_users) == 1 && mymm) {
flush_tlb_mm(mm);
__get_cpu_var(ptcstats).change_rid++;
preempt_enable();
@@ -241,11 +246,13 @@
for_each_node_mask(cnode, nodes_flushed)
nasids[nix++] = cnodeid_to_nasid(cnode);
+ rr_value = (mm->context << 3) | REGION_NUMBER(start);
+
shub1 = is_shub1();
if (shub1) {
data0 = (1UL << SH1_PTC_0_A_SHFT) |
(nbits << SH1_PTC_0_PS_SHFT) |
- ((ia64_get_rr(start) >> 8) << SH1_PTC_0_RID_SHFT) |
+ (rr_value << SH1_PTC_0_RID_SHFT) |
(1UL << SH1_PTC_0_START_SHFT);
ptc0 = (long *)GLOBAL_MMR_PHYS_ADDR(0, SH1_PTC_0);
ptc1 = (long *)GLOBAL_MMR_PHYS_ADDR(0, SH1_PTC_1);
@@ -254,7 +261,7 @@
(nbits << SH2_PTC_PS_SHFT) |
(1UL << SH2_PTC_START_SHFT);
ptc0 = (long *)GLOBAL_MMR_PHYS_ADDR(0, SH2_PTC +
- ((ia64_get_rr(start) >> 8) << SH2_PTC_RID_SHFT) );
+ (rr_value << SH2_PTC_RID_SHFT));
ptc1 = NULL;
}
@@ -275,7 +282,7 @@
data0 = (data0 & ~SH2_PTC_ADDR_MASK) | (start & SH2_PTC_ADDR_MASK);
for (i = 0; i < nix; i++) {
nasid = nasids[i];
- if ((!(sn2_ptctest & 3)) && unlikely(nasid == mynasid)) {
+ if ((!(sn2_ptctest & 3)) && unlikely(nasid == mynasid && mymm)) {
ia64_ptcga(start, nbits << 2);
ia64_srlz_i();
} else {
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread* RE: [ PATCH ] - Avoid slow TLB purges on SGI Altix systems
@ 2005-10-27 16:01 Luck, Tony
2005-10-27 18:31 ` Dean Roe
0 siblings, 1 reply; 6+ messages in thread
From: Luck, Tony @ 2005-10-27 16:01 UTC (permalink / raw)
To: Dean Roe; +Cc: linux-ia64, linux-mm
- if (mm != current->active_mm) {
- /* this does happen, but perhaps it's not worth optimizing for? */
-#ifdef CONFIG_SMP
- flush_tlb_all();
-#else
- mm->context = 0;
-#endif
- return;
- }
Your patch moves this secion of code up to ia64_global_tlb_purge(),
but the new code that is added there doesn't include the UP case
where mm->context is set to zero.
-Tony
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [ PATCH ] - Avoid slow TLB purges on SGI Altix systems
2005-10-27 16:01 [ PATCH ] " Luck, Tony
@ 2005-10-27 18:31 ` Dean Roe
0 siblings, 0 replies; 6+ messages in thread
From: Dean Roe @ 2005-10-27 18:31 UTC (permalink / raw)
To: Luck, Tony; +Cc: Dean Roe, linux-ia64, linux-mm
On Thu, Oct 27, 2005 at 09:01:53AM -0700, Luck, Tony wrote:
> - if (mm != current->active_mm) {
> - /* this does happen, but perhaps it's not worth optimizing for? */
> -#ifdef CONFIG_SMP
> - flush_tlb_all();
> -#else
> - mm->context = 0;
> -#endif
> - return;
> - }
>
> Your patch moves this secion of code up to ia64_global_tlb_purge(),
> but the new code that is added there doesn't include the UP case
> where mm->context is set to zero.
>
> -Tony
>
flush_tlb_range() only calls platform_global_tlb_purge() for CONFIG_SMP,
so there's no point in having that code in ia64_global_tlb_purge().
Dean
--
Dean Roe
Silicon Graphics, Inc.
roe@sgi.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [ PATCH ] - Avoid slow TLB purges on SGI Altix systems
@ 2005-10-14 19:55 Dean Roe
0 siblings, 0 replies; 6+ messages in thread
From: Dean Roe @ 2005-10-14 19:55 UTC (permalink / raw)
To: tony.luck; +Cc: linux-ia64, linux-mm
flush_tlb_all() can be a scaling issue on large SGI Altix systems
since it uses the global call_lock and always executes on all cpus.
When a process enters flush_tlb_range() to purge TLBs for another
process, it is possible to avoid flush_tlb_all() and instead allow
sn2_global_tlb_purge() to purge TLBs only where necessary.
This patch modifies flush_tlb_range() so that this case can be handled
by platform TLB purge functions and updates ia64_global_tlb_purge()
accordingly. sn2_global_tlb_purge() now calculates the region register
value from the mm argument introduced with this patch.
Signed-off-by: Dean Roe <roe@sgi.com>
Index: linux-ia64-test/include/asm-ia64/machvec.h
===================================================================
--- linux-ia64-test.orig/include/asm-ia64/machvec.h
+++ linux-ia64-test/include/asm-ia64/machvec.h
@@ -26,7 +26,7 @@
typedef void ia64_mv_irq_init_t (void);
typedef void ia64_mv_send_ipi_t (int, int, int, int);
typedef void ia64_mv_timer_interrupt_t (int, void *, struct pt_regs *);
-typedef void ia64_mv_global_tlb_purge_t (unsigned long, unsigned long, unsigned long);
+typedef void ia64_mv_global_tlb_purge_t (struct mm_struct *, unsigned long, unsigned long, unsigned long);
typedef void ia64_mv_tlb_migrate_finish_t (struct mm_struct *);
typedef unsigned int ia64_mv_local_vector_to_irq (u8);
typedef char *ia64_mv_pci_get_legacy_mem_t (struct pci_bus *);
Index: linux-ia64-test/arch/ia64/mm/tlb.c
===================================================================
--- linux-ia64-test.orig/arch/ia64/mm/tlb.c
+++ linux-ia64-test/arch/ia64/mm/tlb.c
@@ -86,10 +86,16 @@
}
void
-ia64_global_tlb_purge (unsigned long start, unsigned long end, unsigned long nbits)
+ia64_global_tlb_purge (struct mm_struct *mm, unsigned long start, unsigned long end, unsigned long nbits)
{
static DEFINE_SPINLOCK(ptcg_lock);
+ if (mm != current->active_mm) {
+ /* this does happen, but perhaps it's not worth optimizing for? */
+ flush_tlb_all();
+ return;
+ }
+
/* HW requires global serialization of ptc.ga. */
spin_lock(&ptcg_lock);
{
@@ -135,16 +141,6 @@
unsigned long size = end - start;
unsigned long nbits;
- if (mm != current->active_mm) {
- /* this does happen, but perhaps it's not worth optimizing for? */
-#ifdef CONFIG_SMP
- flush_tlb_all();
-#else
- mm->context = 0;
-#endif
- return;
- }
-
nbits = ia64_fls(size + 0xfff);
while (unlikely (((1UL << nbits) & purge.mask) == 0) && (nbits < purge.max_bits))
++nbits;
@@ -153,7 +149,7 @@
start &= ~((1UL << nbits) - 1);
# ifdef CONFIG_SMP
- platform_global_tlb_purge(start, end, nbits);
+ platform_global_tlb_purge(mm, start, end, nbits);
# else
do {
ia64_ptcl(start, (nbits<<2));
Index: linux-ia64-test/arch/ia64/sn/kernel/sn2/sn2_smp.c
===================================================================
--- linux-ia64-test.orig/arch/ia64/sn/kernel/sn2/sn2_smp.c
+++ linux-ia64-test/arch/ia64/sn/kernel/sn2/sn2_smp.c
@@ -177,6 +177,7 @@
/**
* sn2_global_tlb_purge - globally purge translation cache of virtual address range
+ * @mm: mm_struct containing virtual address range
* @start: start of virtual address range
* @end: end of virtual address range
* @nbits: specifies number of bytes to purge per instruction (num = 1<<(nbits & 0xfc))
@@ -188,21 +189,22 @@
* - cpu_vm_mask is a bit mask that indicates which cpus have loaded the context.
* - cpu_vm_mask is converted into a nodemask of the nodes containing the
* cpus in cpu_vm_mask.
- * - if only one bit is set in cpu_vm_mask & it is the current cpu,
- * then only the local TLB needs to be flushed. This flushing can be done
- * using ptc.l. This is the common case & avoids the global spinlock.
+ * - if only one bit is set in cpu_vm_mask & it is the current cpu & the
+ * process is purging its own virtual address range, then only the
+ * local TLB needs to be flushed. This flushing can be done using
+ * ptc.l. This is the common case & avoids the global spinlock.
* - if multiple cpus have loaded the context, then flushing has to be
* done with ptc.g/MMRs under protection of the global ptc_lock.
*/
void
-sn2_global_tlb_purge(unsigned long start, unsigned long end,
- unsigned long nbits)
+sn2_global_tlb_purge(struct mm_struct *mm, unsigned long start,
+ unsigned long end, unsigned long nbits)
{
int i, opt, shub1, cnode, mynasid, cpu, lcpu = 0, nasid, flushed = 0;
+ int mymm = (mm == current->active_mm);
volatile unsigned long *ptc0, *ptc1;
- unsigned long itc, itc2, flags, data0 = 0, data1 = 0;
- struct mm_struct *mm = current->active_mm;
+ unsigned long itc, itc2, flags, data0 = 0, data1 = 0, rr_value;
short nasids[MAX_NUMNODES], nix;
nodemask_t nodes_flushed;
@@ -216,9 +218,12 @@
i++;
}
+ if (i == 0)
+ return;
+
preempt_disable();
- if (likely(i == 1 && lcpu == smp_processor_id())) {
+ if (likely(i == 1 && lcpu == smp_processor_id() && mymm)) {
do {
ia64_ptcl(start, nbits << 2);
start += (1UL << nbits);
@@ -229,7 +234,7 @@
return;
}
- if (atomic_read(&mm->mm_users) == 1) {
+ if (atomic_read(&mm->mm_users) == 1 && mymm) {
flush_tlb_mm(mm);
__get_cpu_var(ptcstats).change_rid++;
preempt_enable();
@@ -241,11 +246,13 @@
for_each_node_mask(cnode, nodes_flushed)
nasids[nix++] = cnodeid_to_nasid(cnode);
+ rr_value = (mm->context << 3) | REGION_NUMBER(start);
+
shub1 = is_shub1();
if (shub1) {
data0 = (1UL << SH1_PTC_0_A_SHFT) |
(nbits << SH1_PTC_0_PS_SHFT) |
- ((ia64_get_rr(start) >> 8) << SH1_PTC_0_RID_SHFT) |
+ (rr_value << SH1_PTC_0_RID_SHFT) |
(1UL << SH1_PTC_0_START_SHFT);
ptc0 = (long *)GLOBAL_MMR_PHYS_ADDR(0, SH1_PTC_0);
ptc1 = (long *)GLOBAL_MMR_PHYS_ADDR(0, SH1_PTC_1);
@@ -254,7 +261,7 @@
(nbits << SH2_PTC_PS_SHFT) |
(1UL << SH2_PTC_START_SHFT);
ptc0 = (long *)GLOBAL_MMR_PHYS_ADDR(0, SH2_PTC +
- ((ia64_get_rr(start) >> 8) << SH2_PTC_RID_SHFT) );
+ (rr_value << SH2_PTC_RID_SHFT));
ptc1 = NULL;
}
@@ -275,7 +282,7 @@
data0 = (data0 & ~SH2_PTC_ADDR_MASK) | (start & SH2_PTC_ADDR_MASK);
for (i = 0; i < nix; i++) {
nasid = nasids[i];
- if ((!(sn2_ptctest & 3)) && unlikely(nasid == mynasid)) {
+ if ((!(sn2_ptctest & 3)) && unlikely(nasid == mynasid && mymm)) {
ia64_ptcga(start, nbits << 2);
ia64_srlz_i();
} else {
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2005-10-27 20:41 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2005-10-27 18:41 [ PATCH ] - Avoid slow TLB purges on SGI Altix systems Luck, Tony
2005-10-27 18:44 ` Dean Roe
-- strict thread matches above, loose matches on Subject: below --
2005-10-27 20:41 [PATCH] " Dean Roe
2005-10-27 16:01 [ PATCH ] " Luck, Tony
2005-10-27 18:31 ` Dean Roe
2005-10-14 19:55 Dean Roe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox