* [PATCH 0/3] on do_page_fault() and *copy*_inatomic
@ 2006-11-01 11:44 Peter Zijlstra
2006-11-01 11:44 ` [PATCH 1/3] mm: arch do_page_fault() vs in_atomic() Peter Zijlstra
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Peter Zijlstra @ 2006-11-01 11:44 UTC (permalink / raw)
To: linux-kernel, Andrew Morton
Cc: Nick Piggin, Peter Zijlstra, linux-arch, linux-mm
In light of the recent work on fault handlers and generic_file_buffered_write()
I've gone over some of the arch specific stuff that supports this work.
The following three patches are ready for inclusion IMHO, please apply.
The first patch fixes up some arch fault handlers to respect the
'take no locks in atomic context' rule; this also fixes CONFIG_PREEMPT bugs
on those platforms.
The second patch introduces pagefault_{disable,enable}() - an abtraction that
replaces the now open coded {inc,dec}_preempt_count() calls when we mean to
create atomic pagefault scope. The added barrier() calls in the new
primitives might fix some CONFIG_PREEMPT bugs.
The third patch make k{,un}map_atomic denote an atomic pagefault scope. All
non-trivial implementation already do this, and this allows us to rely on that
in generic. This might also fix some bugs where people already assumed this.
Peter
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 1/3] mm: arch do_page_fault() vs in_atomic()
2006-11-01 11:44 [PATCH 0/3] on do_page_fault() and *copy*_inatomic Peter Zijlstra
@ 2006-11-01 11:44 ` Peter Zijlstra
2006-11-01 11:44 ` [PATCH 2/3] mm: pagefault_{disable,enable}() Peter Zijlstra
2006-11-01 11:44 ` [PATCH 3/3] mm: k{,um}map_atomic() vs in_atomic() Peter Zijlstra
2 siblings, 0 replies; 4+ messages in thread
From: Peter Zijlstra @ 2006-11-01 11:44 UTC (permalink / raw)
To: linux-kernel, Andrew Morton
Cc: Nick Piggin, Peter Zijlstra, linux-arch, linux-mm
[-- Attachment #1: inatomic_do_page_fault.patch --]
[-- Type: text/plain, Size: 4925 bytes --]
In light of the recent pagefault and filemap_copy_from_user work I've
gone through all the arch pagefault handlers to make sure the
inc_preempt_count() 'feature' works as expected.
Several sections of code (including the new filemap_copy_from_user) rely
on the fact that faults do not take locks under increased preempt count.
arch/x86_64 - good
arch/powerpc - good
arch/cris - fixed
arch/i386 - good
arch/parisc - fixed
arch/sh - good
arch/sparc - good
arch/s390 - good
arch/m68k - fixed
arch/ppc - good
arch/alpha - fixed
arch/mips - good
arch/sparc64 - good
arch/ia64 - good
arch/arm - fixed
arch/um - good
arch/avr32 - good
arch/h8300 - NA
arch/m32r - good
arch/v850 - good
arch/frv - fixed
arch/m68knommu - NA
arch/arm26 - fixed
arch/sh64 - fixed
arch/xtensa - good
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: Nick Piggin <npiggin@suse.de>
---
arch/alpha/mm/fault.c | 2 +-
arch/arm/mm/fault.c | 2 +-
arch/arm26/mm/fault.c | 2 +-
arch/cris/mm/fault.c | 2 +-
arch/frv/mm/fault.c | 2 +-
arch/m68k/mm/fault.c | 2 +-
arch/parisc/mm/fault.c | 2 +-
arch/sh64/mm/fault.c | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
Index: linux-2.6/arch/alpha/mm/fault.c
===================================================================
--- linux-2.6.orig/arch/alpha/mm/fault.c
+++ linux-2.6/arch/alpha/mm/fault.c
@@ -108,7 +108,7 @@ do_page_fault(unsigned long address, uns
/* If we're in an interrupt context, or have no user context,
we must not take the fault. */
- if (!mm || in_interrupt())
+ if (!mm || in_atomic())
goto no_context;
#ifdef CONFIG_ALPHA_LARGE_VMALLOC
Index: linux-2.6/arch/arm/mm/fault.c
===================================================================
--- linux-2.6.orig/arch/arm/mm/fault.c
+++ linux-2.6/arch/arm/mm/fault.c
@@ -230,7 +230,7 @@ do_page_fault(unsigned long addr, unsign
* If we're in an interrupt or have no user
* context, we must not take the fault..
*/
- if (in_interrupt() || !mm)
+ if (in_atomic() || !mm)
goto no_context;
/*
Index: linux-2.6/arch/arm26/mm/fault.c
===================================================================
--- linux-2.6.orig/arch/arm26/mm/fault.c
+++ linux-2.6/arch/arm26/mm/fault.c
@@ -215,7 +215,7 @@ int do_page_fault(unsigned long addr, un
* If we're in an interrupt or have no user
* context, we must not take the fault..
*/
- if (in_interrupt() || !mm)
+ if (in_atomic() || !mm)
goto no_context;
down_read(&mm->mmap_sem);
Index: linux-2.6/arch/cris/mm/fault.c
===================================================================
--- linux-2.6.orig/arch/cris/mm/fault.c
+++ linux-2.6/arch/cris/mm/fault.c
@@ -232,7 +232,7 @@ do_page_fault(unsigned long address, str
* context, we must not take the fault..
*/
- if (in_interrupt() || !mm)
+ if (in_atomic() || !mm)
goto no_context;
down_read(&mm->mmap_sem);
Index: linux-2.6/arch/frv/mm/fault.c
===================================================================
--- linux-2.6.orig/arch/frv/mm/fault.c
+++ linux-2.6/arch/frv/mm/fault.c
@@ -78,7 +78,7 @@ asmlinkage void do_page_fault(int datamm
* If we're in an interrupt or have no user
* context, we must not take the fault..
*/
- if (in_interrupt() || !mm)
+ if (in_atomic() || !mm)
goto no_context;
down_read(&mm->mmap_sem);
Index: linux-2.6/arch/m68k/mm/fault.c
===================================================================
--- linux-2.6.orig/arch/m68k/mm/fault.c
+++ linux-2.6/arch/m68k/mm/fault.c
@@ -99,7 +99,7 @@ int do_page_fault(struct pt_regs *regs,
* If we're in an interrupt or have no user
* context, we must not take the fault..
*/
- if (in_interrupt() || !mm)
+ if (in_atomic() || !mm)
goto no_context;
down_read(&mm->mmap_sem);
Index: linux-2.6/arch/parisc/mm/fault.c
===================================================================
--- linux-2.6.orig/arch/parisc/mm/fault.c
+++ linux-2.6/arch/parisc/mm/fault.c
@@ -152,7 +152,7 @@ void do_page_fault(struct pt_regs *regs,
const struct exception_table_entry *fix;
unsigned long acc_type;
- if (in_interrupt() || !mm)
+ if (in_atomic() || !mm)
goto no_context;
down_read(&mm->mmap_sem);
Index: linux-2.6/arch/sh64/mm/fault.c
===================================================================
--- linux-2.6.orig/arch/sh64/mm/fault.c
+++ linux-2.6/arch/sh64/mm/fault.c
@@ -154,7 +154,7 @@ asmlinkage void do_page_fault(struct pt_
* If we're in an interrupt or have no user
* context, we must not take the fault..
*/
- if (in_interrupt() || !mm)
+ if (in_atomic() || !mm)
goto no_context;
/* TLB misses upon some cache flushes get done under cli() */
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 2/3] mm: pagefault_{disable,enable}()
2006-11-01 11:44 [PATCH 0/3] on do_page_fault() and *copy*_inatomic Peter Zijlstra
2006-11-01 11:44 ` [PATCH 1/3] mm: arch do_page_fault() vs in_atomic() Peter Zijlstra
@ 2006-11-01 11:44 ` Peter Zijlstra
2006-11-01 11:44 ` [PATCH 3/3] mm: k{,um}map_atomic() vs in_atomic() Peter Zijlstra
2 siblings, 0 replies; 4+ messages in thread
From: Peter Zijlstra @ 2006-11-01 11:44 UTC (permalink / raw)
To: linux-kernel, Andrew Morton
Cc: Nick Piggin, Peter Zijlstra, linux-arch, linux-mm
[-- Attachment #1: pagefault_disable.patch --]
[-- Type: text/plain, Size: 16298 bytes --]
Introduce pagefault_{disable,enable}() and use these where previously
we did manual preempt increments/decrements to make the pagefault handler
do the atomic thing.
Currently they still rely on the increased preempt count, but do not rely
on the disabled preemption, this might go away in the future.
(NOTE: the extra barrier() in pagefault_disable might fix some holes on
machines which have too many registers for their own good)
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: Nick Piggin <npiggin@suse.de>
---
arch/frv/kernel/futex.c | 4 ++--
arch/i386/mm/highmem.c | 10 ++++------
arch/mips/mm/highmem.c | 10 ++++------
arch/s390/lib/uaccess_std.c | 4 ++--
arch/sparc/mm/highmem.c | 8 +++-----
include/asm-frv/highmem.h | 5 ++---
include/asm-generic/futex.h | 4 ++--
include/asm-i386/futex.h | 4 ++--
include/asm-ia64/futex.h | 4 ++--
include/asm-mips/futex.h | 4 ++--
include/asm-parisc/futex.h | 4 ++--
include/asm-powerpc/futex.h | 4 ++--
include/asm-ppc/highmem.h | 8 +++-----
include/asm-sparc64/futex.h | 4 ++--
include/asm-x86_64/futex.h | 4 ++--
include/linux/uaccess.h | 39 +++++++++++++++++++++++++++++++++++++--
kernel/futex.c | 28 ++++++++++++++--------------
17 files changed, 87 insertions(+), 61 deletions(-)
Index: linux-2.6/arch/frv/kernel/futex.c
===================================================================
--- linux-2.6.orig/arch/frv/kernel/futex.c
+++ linux-2.6/arch/frv/kernel/futex.c
@@ -200,7 +200,7 @@ int futex_atomic_op_inuser(int encoded_o
if (!access_ok(VERIFY_WRITE, uaddr, sizeof(int)))
return -EFAULT;
- inc_preempt_count();
+ pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -223,7 +223,7 @@ int futex_atomic_op_inuser(int encoded_o
break;
}
- dec_preempt_count();
+ pagefault_enable();
if (!ret) {
switch (cmp) {
Index: linux-2.6/arch/i386/mm/highmem.c
===================================================================
--- linux-2.6.orig/arch/i386/mm/highmem.c
+++ linux-2.6/arch/i386/mm/highmem.c
@@ -32,7 +32,7 @@ void *kmap_atomic(struct page *page, enu
unsigned long vaddr;
/* even !CONFIG_PREEMPT needs this, for in_atomic in do_page_fault */
- inc_preempt_count();
+ pagefault_disable();
if (!PageHighMem(page))
return page_address(page);
@@ -52,8 +52,7 @@ void kunmap_atomic(void *kvaddr, enum km
#ifdef CONFIG_DEBUG_HIGHMEM
if (vaddr >= PAGE_OFFSET && vaddr < (unsigned long)high_memory) {
- dec_preempt_count();
- preempt_check_resched();
+ pagefault_enable();
return;
}
@@ -68,8 +67,7 @@ void kunmap_atomic(void *kvaddr, enum km
*/
kpte_clear_flush(kmap_pte-idx, vaddr);
- dec_preempt_count();
- preempt_check_resched();
+ pagefault_enable();
}
/* This is the same as kmap_atomic() but can map memory that doesn't
@@ -80,7 +78,7 @@ void *kmap_atomic_pfn(unsigned long pfn,
enum fixed_addresses idx;
unsigned long vaddr;
- inc_preempt_count();
+ pagefault_disable();
idx = type + KM_TYPE_NR*smp_processor_id();
vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
Index: linux-2.6/arch/mips/mm/highmem.c
===================================================================
--- linux-2.6.orig/arch/mips/mm/highmem.c
+++ linux-2.6/arch/mips/mm/highmem.c
@@ -39,7 +39,7 @@ void *__kmap_atomic(struct page *page, e
unsigned long vaddr;
/* even !CONFIG_PREEMPT needs this, for in_atomic in do_page_fault */
- inc_preempt_count();
+ pagefault_disable();
if (!PageHighMem(page))
return page_address(page);
@@ -62,8 +62,7 @@ void __kunmap_atomic(void *kvaddr, enum
enum fixed_addresses idx = type + KM_TYPE_NR*smp_processor_id();
if (vaddr < FIXADDR_START) { // FIXME
- dec_preempt_count();
- preempt_check_resched();
+ pagefault_enable();
return;
}
@@ -78,8 +77,7 @@ void __kunmap_atomic(void *kvaddr, enum
local_flush_tlb_one(vaddr);
#endif
- dec_preempt_count();
- preempt_check_resched();
+ pagefault_enable();
}
#ifndef CONFIG_LIMITED_DMA
@@ -92,7 +90,7 @@ void *kmap_atomic_pfn(unsigned long pfn,
enum fixed_addresses idx;
unsigned long vaddr;
- inc_preempt_count();
+ pagefault_disable();
idx = type + KM_TYPE_NR*smp_processor_id();
vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
Index: linux-2.6/arch/s390/lib/uaccess_std.c
===================================================================
--- linux-2.6.orig/arch/s390/lib/uaccess_std.c
+++ linux-2.6/arch/s390/lib/uaccess_std.c
@@ -295,7 +295,7 @@ int futex_atomic_op(int op, int __user *
{
int oldval = 0, newval, ret;
- inc_preempt_count();
+ pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -321,7 +321,7 @@ int futex_atomic_op(int op, int __user *
default:
ret = -ENOSYS;
}
- dec_preempt_count();
+ pagefault_enable();
*old = oldval;
return ret;
}
Index: linux-2.6/arch/sparc/mm/highmem.c
===================================================================
--- linux-2.6.orig/arch/sparc/mm/highmem.c
+++ linux-2.6/arch/sparc/mm/highmem.c
@@ -35,7 +35,7 @@ void *kmap_atomic(struct page *page, enu
unsigned long vaddr;
/* even !CONFIG_PREEMPT needs this, for in_atomic in do_page_fault */
- inc_preempt_count();
+ pagefault_disable();
if (!PageHighMem(page))
return page_address(page);
@@ -70,8 +70,7 @@ void kunmap_atomic(void *kvaddr, enum km
unsigned long idx = type + KM_TYPE_NR*smp_processor_id();
if (vaddr < FIXADDR_START) { // FIXME
- dec_preempt_count();
- preempt_check_resched();
+ pagefault_enable();
return;
}
@@ -97,8 +96,7 @@ void kunmap_atomic(void *kvaddr, enum km
#endif
#endif
- dec_preempt_count();
- preempt_check_resched();
+ pagefault_enable();
}
/* We may be fed a pagetable here by ptep_to_xxx and others. */
Index: linux-2.6/include/asm-frv/highmem.h
===================================================================
--- linux-2.6.orig/include/asm-frv/highmem.h
+++ linux-2.6/include/asm-frv/highmem.h
@@ -115,7 +115,7 @@ static inline void *kmap_atomic(struct p
{
unsigned long paddr;
- inc_preempt_count();
+ pagefault_disable();
paddr = page_to_phys(page);
switch (type) {
@@ -170,8 +170,7 @@ static inline void kunmap_atomic(void *k
default:
BUG();
}
- dec_preempt_count();
- preempt_check_resched();
+ pagefault_enable();
}
#endif /* !__ASSEMBLY__ */
Index: linux-2.6/include/asm-generic/futex.h
===================================================================
--- linux-2.6.orig/include/asm-generic/futex.h
+++ linux-2.6/include/asm-generic/futex.h
@@ -21,7 +21,7 @@ futex_atomic_op_inuser (int encoded_op,
if (! access_ok (VERIFY_WRITE, uaddr, sizeof(int)))
return -EFAULT;
- inc_preempt_count();
+ pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -33,7 +33,7 @@ futex_atomic_op_inuser (int encoded_op,
ret = -ENOSYS;
}
- dec_preempt_count();
+ pagefault_enable();
if (!ret) {
switch (cmp) {
Index: linux-2.6/include/asm-i386/futex.h
===================================================================
--- linux-2.6.orig/include/asm-i386/futex.h
+++ linux-2.6/include/asm-i386/futex.h
@@ -56,7 +56,7 @@ futex_atomic_op_inuser (int encoded_op,
if (! access_ok (VERIFY_WRITE, uaddr, sizeof(int)))
return -EFAULT;
- inc_preempt_count();
+ pagefault_disable();
if (op == FUTEX_OP_SET)
__futex_atomic_op1("xchgl %0, %2", ret, oldval, uaddr, oparg);
@@ -88,7 +88,7 @@ futex_atomic_op_inuser (int encoded_op,
}
}
- dec_preempt_count();
+ pagefault_enable();
if (!ret) {
switch (cmp) {
Index: linux-2.6/include/asm-ia64/futex.h
===================================================================
--- linux-2.6.orig/include/asm-ia64/futex.h
+++ linux-2.6/include/asm-ia64/futex.h
@@ -59,7 +59,7 @@ futex_atomic_op_inuser (int encoded_op,
if (! access_ok (VERIFY_WRITE, uaddr, sizeof(int)))
return -EFAULT;
- inc_preempt_count();
+ pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -83,7 +83,7 @@ futex_atomic_op_inuser (int encoded_op,
ret = -ENOSYS;
}
- dec_preempt_count();
+ pagefault_enable();
if (!ret) {
switch (cmp) {
Index: linux-2.6/include/asm-mips/futex.h
===================================================================
--- linux-2.6.orig/include/asm-mips/futex.h
+++ linux-2.6/include/asm-mips/futex.h
@@ -86,7 +86,7 @@ futex_atomic_op_inuser (int encoded_op,
if (! access_ok (VERIFY_WRITE, uaddr, sizeof(int)))
return -EFAULT;
- inc_preempt_count();
+ pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -113,7 +113,7 @@ futex_atomic_op_inuser (int encoded_op,
ret = -ENOSYS;
}
- dec_preempt_count();
+ pagefault_enable();
if (!ret) {
switch (cmp) {
Index: linux-2.6/include/asm-parisc/futex.h
===================================================================
--- linux-2.6.orig/include/asm-parisc/futex.h
+++ linux-2.6/include/asm-parisc/futex.h
@@ -21,7 +21,7 @@ futex_atomic_op_inuser (int encoded_op,
if (! access_ok (VERIFY_WRITE, uaddr, sizeof(int)))
return -EFAULT;
- inc_preempt_count();
+ pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -33,7 +33,7 @@ futex_atomic_op_inuser (int encoded_op,
ret = -ENOSYS;
}
- dec_preempt_count();
+ pagefault_enable();
if (!ret) {
switch (cmp) {
Index: linux-2.6/include/asm-powerpc/futex.h
===================================================================
--- linux-2.6.orig/include/asm-powerpc/futex.h
+++ linux-2.6/include/asm-powerpc/futex.h
@@ -43,7 +43,7 @@ static inline int futex_atomic_op_inuser
if (! access_ok (VERIFY_WRITE, uaddr, sizeof(int)))
return -EFAULT;
- inc_preempt_count();
+ pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -65,7 +65,7 @@ static inline int futex_atomic_op_inuser
ret = -ENOSYS;
}
- dec_preempt_count();
+ pagefault_enable();
if (!ret) {
switch (cmp) {
Index: linux-2.6/include/asm-ppc/highmem.h
===================================================================
--- linux-2.6.orig/include/asm-ppc/highmem.h
+++ linux-2.6/include/asm-ppc/highmem.h
@@ -79,7 +79,7 @@ static inline void *kmap_atomic(struct p
unsigned long vaddr;
/* even !CONFIG_PREEMPT needs this, for in_atomic in do_page_fault */
- inc_preempt_count();
+ pagefault_disable();
if (!PageHighMem(page))
return page_address(page);
@@ -101,8 +101,7 @@ static inline void kunmap_atomic(void *k
unsigned int idx = type + KM_TYPE_NR*smp_processor_id();
if (vaddr < KMAP_FIX_BEGIN) { // FIXME
- dec_preempt_count();
- preempt_check_resched();
+ pagefault_enable();
return;
}
@@ -115,8 +114,7 @@ static inline void kunmap_atomic(void *k
pte_clear(&init_mm, vaddr, kmap_pte+idx);
flush_tlb_page(NULL, vaddr);
#endif
- dec_preempt_count();
- preempt_check_resched();
+ pagefault_enable();
}
static inline struct page *kmap_atomic_to_page(void *ptr)
Index: linux-2.6/include/asm-sparc64/futex.h
===================================================================
--- linux-2.6.orig/include/asm-sparc64/futex.h
+++ linux-2.6/include/asm-sparc64/futex.h
@@ -45,7 +45,7 @@ static inline int futex_atomic_op_inuser
if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
oparg = 1 << oparg;
- inc_preempt_count();
+ pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -67,7 +67,7 @@ static inline int futex_atomic_op_inuser
ret = -ENOSYS;
}
- dec_preempt_count();
+ pagefault_enable();
if (!ret) {
switch (cmp) {
Index: linux-2.6/include/asm-x86_64/futex.h
===================================================================
--- linux-2.6.orig/include/asm-x86_64/futex.h
+++ linux-2.6/include/asm-x86_64/futex.h
@@ -55,7 +55,7 @@ futex_atomic_op_inuser (int encoded_op,
if (! access_ok (VERIFY_WRITE, uaddr, sizeof(int)))
return -EFAULT;
- inc_preempt_count();
+ pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -78,7 +78,7 @@ futex_atomic_op_inuser (int encoded_op,
ret = -ENOSYS;
}
- dec_preempt_count();
+ pagefault_enable();
if (!ret) {
switch (cmp) {
Index: linux-2.6/include/linux/uaccess.h
===================================================================
--- linux-2.6.orig/include/linux/uaccess.h
+++ linux-2.6/include/linux/uaccess.h
@@ -1,8 +1,43 @@
#ifndef __LINUX_UACCESS_H__
#define __LINUX_UACCESS_H__
+#include <linux/preempt.h>
#include <asm/uaccess.h>
+/*
+ * These routines enable/disable the pagefault handler in that
+ * it will not take any locks and go straight to the fixup table.
+ *
+ * They have great resemblance to the preempt_disable/enable calls
+ * and in fact they are identical; this is because currently there is
+ * no other way to make the pagefault handlers do this. So we do
+ * disable preemption but we don't necessarily care about that.
+ */
+static inline void pagefault_disable(void)
+{
+ inc_preempt_count();
+ /*
+ * make sure to have issued the store before a pagefault
+ * can hit.
+ */
+ barrier();
+}
+
+static inline void pagefault_enable(void)
+{
+ /*
+ * make sure to issue those last loads/stores before enabling
+ * the pagefault handler again.
+ */
+ barrier();
+ dec_preempt_count();
+ /*
+ * make sure we do..
+ */
+ barrier();
+ preempt_check_resched();
+}
+
#ifndef ARCH_HAS_NOCACHE_UACCESS
static inline unsigned long __copy_from_user_inatomic_nocache(void *to,
@@ -35,9 +70,9 @@ static inline unsigned long __copy_from_
({ \
long ret; \
\
- inc_preempt_count(); \
+ pagefault_disable(); \
ret = __get_user(retval, addr); \
- dec_preempt_count(); \
+ pagefault_enable(); \
ret; \
})
Index: linux-2.6/kernel/futex.c
===================================================================
--- linux-2.6.orig/kernel/futex.c
+++ linux-2.6/kernel/futex.c
@@ -282,9 +282,9 @@ static inline int get_futex_value_locked
{
int ret;
- inc_preempt_count();
+ pagefault_disable();
ret = __copy_from_user_inatomic(dest, from, sizeof(u32));
- dec_preempt_count();
+ pagefault_enable();
return ret ? -EFAULT : 0;
}
@@ -585,9 +585,9 @@ static int wake_futex_pi(u32 __user *uad
if (!(uval & FUTEX_OWNER_DIED)) {
newval = FUTEX_WAITERS | new_owner->pid;
- inc_preempt_count();
+ pagefault_disable();
curval = futex_atomic_cmpxchg_inatomic(uaddr, uval, newval);
- dec_preempt_count();
+ pagefault_enable();
if (curval == -EFAULT)
return -EFAULT;
if (curval != uval)
@@ -618,9 +618,9 @@ static int unlock_futex_pi(u32 __user *u
* There is no waiter, so we unlock the futex. The owner died
* bit has not to be preserved here. We are the owner:
*/
- inc_preempt_count();
+ pagefault_disable();
oldval = futex_atomic_cmpxchg_inatomic(uaddr, uval, 0);
- dec_preempt_count();
+ pagefault_enable();
if (oldval == -EFAULT)
return oldval;
@@ -1158,9 +1158,9 @@ static int futex_lock_pi(u32 __user *uad
*/
newval = current->pid;
- inc_preempt_count();
+ pagefault_disable();
curval = futex_atomic_cmpxchg_inatomic(uaddr, 0, newval);
- dec_preempt_count();
+ pagefault_enable();
if (unlikely(curval == -EFAULT))
goto uaddr_faulted;
@@ -1183,9 +1183,9 @@ static int futex_lock_pi(u32 __user *uad
uval = curval;
newval = uval | FUTEX_WAITERS;
- inc_preempt_count();
+ pagefault_disable();
curval = futex_atomic_cmpxchg_inatomic(uaddr, uval, newval);
- dec_preempt_count();
+ pagefault_enable();
if (unlikely(curval == -EFAULT))
goto uaddr_faulted;
@@ -1215,10 +1215,10 @@ static int futex_lock_pi(u32 __user *uad
newval = current->pid |
FUTEX_OWNER_DIED | FUTEX_WAITERS;
- inc_preempt_count();
+ pagefault_disable();
curval = futex_atomic_cmpxchg_inatomic(uaddr,
uval, newval);
- dec_preempt_count();
+ pagefault_enable();
if (unlikely(curval == -EFAULT))
goto uaddr_faulted;
@@ -1390,9 +1390,9 @@ retry_locked:
* anyone else up:
*/
if (!(uval & FUTEX_OWNER_DIED)) {
- inc_preempt_count();
+ pagefault_disable();
uval = futex_atomic_cmpxchg_inatomic(uaddr, current->pid, 0);
- dec_preempt_count();
+ pagefault_enable();
}
if (unlikely(uval == -EFAULT))
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 3/3] mm: k{,um}map_atomic() vs in_atomic()
2006-11-01 11:44 [PATCH 0/3] on do_page_fault() and *copy*_inatomic Peter Zijlstra
2006-11-01 11:44 ` [PATCH 1/3] mm: arch do_page_fault() vs in_atomic() Peter Zijlstra
2006-11-01 11:44 ` [PATCH 2/3] mm: pagefault_{disable,enable}() Peter Zijlstra
@ 2006-11-01 11:44 ` Peter Zijlstra
2 siblings, 0 replies; 4+ messages in thread
From: Peter Zijlstra @ 2006-11-01 11:44 UTC (permalink / raw)
To: linux-kernel, Andrew Morton
Cc: Nick Piggin, Peter Zijlstra, linux-arch, linux-mm
[-- Attachment #1: kmap_atomic_generic.patch --]
[-- Type: text/plain, Size: 2401 bytes --]
Make kmap_atomic/kunmap_atomic denote a pagefault disabled scope. All
non trivial implementations already do this anyway.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: Nick Piggin <npiggin@suse.de>
---
include/asm-mips/highmem.h | 10 ++++++++--
include/linux/highmem.h | 8 +++++---
2 files changed, 13 insertions(+), 5 deletions(-)
Index: linux-2.6/include/asm-mips/highmem.h
===================================================================
--- linux-2.6.orig/include/asm-mips/highmem.h
+++ linux-2.6/include/asm-mips/highmem.h
@@ -21,6 +21,7 @@
#include <linux/init.h>
#include <linux/interrupt.h>
+#include <linux/uaccess.h>
#include <asm/kmap_types.h>
/* undef for production */
@@ -70,11 +71,16 @@ static inline void *kmap(struct page *pa
static inline void *kmap_atomic(struct page *page, enum km_type type)
{
+ pagefault_disable();
return page_address(page);
}
-static inline void kunmap_atomic(void *kvaddr, enum km_type type) { }
-#define kmap_atomic_pfn(pfn, idx) page_address(pfn_to_page(pfn))
+static inline void kunmap_atomic(void *kvaddr, enum km_type type)
+{
+ pagefault_enable();
+}
+
+#define kmap_atomic_pfn(pfn, idx) kmap_atomic(pfn_to_page(pfn), (idx))
#define kmap_atomic_to_page(ptr) virt_to_page(ptr)
Index: linux-2.6/include/linux/highmem.h
===================================================================
--- linux-2.6.orig/include/linux/highmem.h
+++ linux-2.6/include/linux/highmem.h
@@ -3,6 +3,7 @@
#include <linux/fs.h>
#include <linux/mm.h>
+#include <linux/uaccess.h>
#include <asm/cacheflush.h>
@@ -41,9 +42,10 @@ static inline void *kmap(struct page *pa
#define kunmap(page) do { (void) (page); } while (0)
-#define kmap_atomic(page, idx) page_address(page)
-#define kunmap_atomic(addr, idx) do { } while (0)
-#define kmap_atomic_pfn(pfn, idx) page_address(pfn_to_page(pfn))
+#define kmap_atomic(page, idx) \
+ ({ pagefault_disable(); page_address(page); })
+#define kunmap_atomic(addr, idx) do { pagefault_enable(); } while (0)
+#define kmap_atomic_pfn(pfn, idx) kmap_atomic(pfn_to_page(pfn), (idx))
#define kmap_atomic_to_page(ptr) virt_to_page(ptr)
#endif
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2006-11-01 11:44 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-11-01 11:44 [PATCH 0/3] on do_page_fault() and *copy*_inatomic Peter Zijlstra
2006-11-01 11:44 ` [PATCH 1/3] mm: arch do_page_fault() vs in_atomic() Peter Zijlstra
2006-11-01 11:44 ` [PATCH 2/3] mm: pagefault_{disable,enable}() Peter Zijlstra
2006-11-01 11:44 ` [PATCH 3/3] mm: k{,um}map_atomic() vs in_atomic() Peter Zijlstra
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox