[1.] One line summary of the problem:
   Large (256MB) CMA allocation fails from time to time when using Intel VT-d

[2.] Full description of the problem/report:

   This is my first bug report so sorry everyone if it may not be complete.
    This is also my first kernel driver, so the issue I'm going to describe may be just because of me and my understanding of linux, but I couldn't find enough information online to solve it.

   I have a development system which consists of:
           - Asus B250 Mining Expert with I7-7700K with 32GB of DDR4 Memory
              - The only reason to use this specific motherboard is because we needed lots of pcie slots for our custom boards.
           - Ubuntu 18.04.03 with a custom compiled 5.0.21 Kernel with the same Ubuntu .config file + DMA CMA support (the .config file is attached)
           - 8 to 16 custom PCI Express 1.0 x1 Boards
           - A simple custom PCI Express Kernel Driver which initializes these boards and allocates 256MB of CMA memory for these boards (They can't do scatter gather)
           - Kernel Command line configured as follows: quiet splash cma=4096M@28G intel_iommu=on
              - I've force the CMA Allocation to the last 4GB of memory and enabled the intel iommu to let our custom PCI boards successfully DMA over.

           When the driver intializes the memory areas it needs for every board, it calls the following code:

           if (dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)))
           {
              printk(KERN_WARNING "ceia_pci: init DMA not supported!\n");
              current_board->dma_support = FALSE;
           } else {

              current_board->dma_support = TRUE;
              current_board->dma_buffer_size = dma_buffer_size;
              current_board->dma_cpu_address = dma_alloc_coherent(&pdev->dev, current_board->dma_buffer_size,
                                                            &current_board->dma_pci_bus_address, GFP_KERNEL);

              if (current_board->dma_cpu_address == NULL) {
                  printk(KERN_WARNING "ceia_pci: init DMA alloc failed! No DMA buffer.\n");
                  current_board->dma_support = FALSE;
              } else {
                 printk(KERN_WARNING "ceia_pci: init DMA alloc Success! DMA buffer size is: %lu MB, %p, %p\n",
                   (current_board->dma_buffer_size / 1024 / 1024), (void *) current_board->dma_pci_bus_address,
                   (void *) current_board->dma_cpu_address);
              }

           }

           From time to time I experience the following error:

[    2.723316] WARNING: CPU: 1 PID: 365 at mm/page_alloc.c:4540 __alloc_pages_nodemask+0x1b9/0x2c0
[    2.723316] Modules linked in: ceia_pci_kernel_driver_throughput(OE+) sunrpc(+) parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic usbhid hid uas usb_storage e1000e r8169 realtek ahci libahci wmi video
[    2.723322] CPU: 1 PID: 365 Comm: systemd-modules Tainted: G           OE     5.0.21ceia-fpga-cma #1
[    2.723323] Hardware name: System manufacturer System Product Name/B250 MINING EXPERT, BIOS 1001 12/13/2017
[    2.723324] RIP: 0010:__alloc_pages_nodemask+0x1b9/0x2c0
[    2.723325] Code: 36 ff ff ff 65 48 8b 04 25 40 5c 01 00 48 05 f8 0b 00 00 41 bf 01 00 00 00 48 89 45 b0 e9 18 ff ff ff 81 e7 00 02 00 00 75 02 <0f> 0b 45 31 ff eb 95 44 8b 6d c0 65 8b 05 f5 d6 7f 7d 89 c0 48 0f
[    2.723326] RSP: 0018:ffffb8cb839178c8 EFLAGS: 00010246
[    2.723327] RAX: 0000000000000000 RBX: ffffffff84293980 RCX: 0000000000000000
[    2.723327] RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
[    2.723328] RBP: ffffb8cb83917920 R08: 0000000000000000 R09: 0000000000000004
[    2.723328] R10: 0000000000000000 R11: 0000000000000001 R12: 00000000006000c0
[    2.723329] R13: 0000000000000010 R14: 0000000000000000 R15: ffffffffc0154708
[    2.723330] FS: 00007fd1a2781940(0000) GS:ffff92a276a40000(0000) knlGS:0000000000000000
[    2.723330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    2.723331] CR2: 00007fd1a26ccef5 CR3: 0000000828c88004 CR4: 00000000003606e0
[    2.723331] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    2.723332] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    2.723332] Call Trace:
[    2.723336] alloc_pages_current+0x6a/0xe0
[    2.723339] intel_alloc_coherent+0xce/0x1c0
[    2.723341] dma_alloc_attrs+0xaa/0xb0
[    2.723343] dma_alloc_coherent+0x40/0x42 [ceia_pci_kernel_driver_throughput]
[    2.723345] initBoard+0x286/0x567 [ceia_pci_kernel_driver_throughput]
[    2.723347] ceia_fpga_probe+0xaf/0x234 [ceia_pci_kernel_driver_throughput]
[    2.723349] local_pci_probe+0x47/0xa0
[    2.723350] pci_device_probe+0x10e/0x1c0
[    2.723352] really_probe+0xfe/0x3f0
[    2.723353] driver_probe_device+0x11a/0x130
[    2.723354] __driver_attach+0x114/0x120
[    2.723356] ? driver_probe_device+0x130/0x130
[    2.723357] ? driver_probe_device+0x130/0x130
[    2.723358] bus_for_each_dev+0x74/0xb0
[    2.723359] ? kmem_cache_alloc_trace+0x151/0x1c0
[    2.723360] driver_attach+0x1e/0x20
[    2.723361] bus_add_driver+0x167/0x260
[    2.723363] ? ceia_fpga_probe+0x234/0x234 [ceia_pci_kernel_driver_throughput]
[    2.723364] driver_register+0x60/0x100
[    2.723365] ? ceia_fpga_probe+0x234/0x234 [ceia_pci_kernel_driver_throughput]
[    2.723366] __pci_register_driver+0x5a/0x60
[    2.723368] ceia_fpga_init+0x184/0xdcc [ceia_pci_kernel_driver_throughput]
[    2.723370] do_one_initcall+0x4a/0x1c9
[    2.723372] ? _cond_resched+0x19/0x40
[    2.723373] ? kmem_cache_alloc_trace+0x151/0x1c0
[    2.723374] do_init_module+0x5f/0x216
[    2.723375] load_module+0x24fe/0x2bd0
[    2.723377] __do_sys_finit_module+0xfc/0x120
[    2.723378] ? __do_sys_finit_module+0xfc/0x120
[    2.723379] __x64_sys_finit_module+0x1a/0x20
[    2.723381] do_syscall_64+0x5a/0x120
[    2.723382] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[    2.723383] RIP: 0033:0x7fd1a22a7839
[    2.723384] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1f f6 2c 00 f7 d8 64 89 01 48
[    2.723384] RSP: 002b:00007ffdffbd0978 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[    2.723385] RAX: ffffffffffffffda RBX: 000055db37447840 RCX: 00007fd1a22a7839
[    2.723386] RDX: 0000000000000000 RSI: 000055db37447950 RDI: 0000000000000005
[    2.723386] RBP: 000055db37447950 R08: 0000000000000000 R09: 0000000000000000
[    2.723387] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000000000
[    2.723387] R13: 000055db374478e0 R14: 0000000000020000 R15: 0000000000000000
[    2.723389] ---[ end trace a693970b6b5d62e6 ]---

I'm not able to reproduce this problem constantly. It happens from time to time. However it seems that a reboot fixes it and that the problem arises after a cold boot (not every cold boot anyway) but this may be misleading, so take it with caution.

This was not happening when we dealt with lot less data and the VT-d was not enabled in the BIOS and the CMA was allocated in the 32bits memory area.

[3.] Keywords (i.e., modules, networking, kernel):
     PCI Express, CMA, VT-d, intel_iommu
[4.] Kernel information
   Kernel 5.0.21 compiled with CMA DMA Support
[4.1.] Kernel version (from /proc/version):
        Linux version 5.0.21ceia-fpga-cma (gmoretti@gmorettipc) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #1 SMP Fri Dec 13 17:53:31 CET 2019
[4.2.] Kernel .config file:
   Attached to the email for bot kernels
[5.] Most recent kernel version which did not have the bug:
[6.] Output of Oops.. message (if applicable) with symbolic information
     resolved (see Documentation/admin-guide/bug-hunting.rst)
[7.] A small shell script or example program which triggers the
     problem (if possible)
[8.] Environment
   Ubuntu 18.04.03
[8.1.] Software (add the output of the ver_linux script here)

Linux ceia-acq-system 5.0.21ceia-fpga-cma #1 SMP Fri Dec 13 17:53:31 CET 2019 x86_64 x86_64 x86_64 GNU/Linux

   GNU Make               4.1
   Binutils               2.30
   Util-linux             2.31.1
   Mount                  2.31.1
   Module-init-tools     24
   E2fsprogs              1.44.1
   Pcmciautils            018
   PPP                    2.4.7
   Nfs-utils              1.3.3
   Linux C Library        2.27
   Dynamic linker (ldd)   2.27
   Linux C++ Library     6.0.25
   Procps                 3.3.12
   Kbd                    2.0.4
   Console-tools          2.0.4
   Sh-utils               8.28
   Udev                   237
   Wireless-tools         30
   Modules Loaded         acpi_pad aesni_intel aes_x86_64 ahci asus_wmi aufs autofs4 bpfilter bridge br_netfilter ceia_pci_kernel_driver_throughput coretemp crc32_pclmul crct10dif_pclmul cryptd crypto_simd drm drm_kms_helper e1000e eeepc_wmi fb_sys_fops ghash_clmulni_intel glue_helper hid hid_generic hid_multitouch i2c_algo_bit i915 intel_cstate intel_powerclamp intel_rapl intel_rapl_perf iptable_filter iptable_nat ip_tables ipt_MASQUERADE irqbypass joydev kvm kvmgt kvm_intel ledtrig_audio libahci libcrc32c llc lp mac_hid mdev mei mei_me mxm_wmi nf_conntrack nf_conntrack_netlink nf_defrag_ipv4 nf_defrag_ipv6 nf_nat nf_nat_ipv4 nfnetlink nls_iso8859_1 overlay parport parport_pc pl2303 ppdev r8169 realtek sch_fq_codel snd snd_hda_codec snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_core snd_hda_intel snd_hwdep snd_pcm snd_rawmidi snd_seq snd_seq_device snd_seq_midi snd_seq_midi_event snd_timer soundcore sparse_keymap stp sunrpc syscopyarea sysfillrect sysimgblt uas usbhid usbserial usb_storage veth vfio vfio_iommu_type1 vfio_mdev video wmi wmi_bmof x86_pkg_temp_thermal xfrm_algo xfrm_user x_tables xt_addrtype xt_conntrack xt_nat xt_tcpudp

[8.2.] Processor information (from /proc/cpuinfo):
   Attached to this email
[8.3.] Module information (from /proc/modules):
   Attached to this email
[8.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
   Attached to this email
[8.5.] PCI information ('lspci -vvv' as root)
   Attached to this email
[8.6.] SCSI information (from /proc/scsi/scsi)
   Attached to this email
[8.7.] Other information that might be relevant to the problem
       (please look in /proc and include all information that you
       think to be relevant):
[X.] Other notes, patches, fixes, workarounds:

--
Sig. Guglielmo Moretti
-----------------------------------
Technical Department
SWE Laboratory
Tel.+39 0575 4181
e-mail:gmoretti@ceia-spa.com

Nota di riservatezza:

Il presente messaggio contiene informazioni da considerarsi strettamente riservate ed è destinato esclusivamente al destinatario/i sopra indicato/i. Chiunque ricevesse questo messaggio per errore o comunque lo leggesse senza esserne legittimato deve intendersi avvertito che qualsiasi forma di comunicazione a terzi o diffusione dei dati, dei fatti e delle notizie apprese è assolutamente vietata nonché perseguita ai sensi di legge ed è pregato di rinviarlo immediatamente al mittente distruggendone l’originale.

Confidentiality notice:

This E-mail message contains information to be deemed strictly confidential and is destined only to the addressee(s) identified above. If anyone else receives this message by mistake or reads it without entitlement is forewarned that keeping, copying, disseminating or distributing this E-mail message to anyone is strictly forbidden and is asked to transmit it immediately to the sender and to erase the original message received.