Thanks Julia. This bug should already be fixed in v7 and later versions: v7 changes - Fix a bug "KASAN: stack-out-of-bounds Read in collapse_file". After copying all pages into the huge page, clear_highpage should use `index` instead of `page->index`. Thanks, Jiaqi On Fri, Dec 23, 2022 at 12:45 AM Julia Lawall wrote: > > See the code lines starting at 2020. > > It looks like if result is still SCAN_SUCCEED, then page has iterated all > the way around the loop and has reached the dummy list head. In that > case, page->index on line 2038 is not likely to be a valid reference. > > julia > > ---------- Forwarded message ---------- > Date: Sun, 13 Nov 2022 20:00:09 +0800 > From: kernel test robot > To: oe-kbuild@lists.linux.dev > Cc: lkp@intel.com, Julia Lawall > Subject: [ammarfaizi2-block:akpm/mm/mm-unstable 137/175] > mm/khugepaged.c:2038:27-31: ERROR: invalid reference to the index variable > of the iterator on line 2026 > > BCC: lkp@intel.com > CC: oe-kbuild-all@lists.linux.dev > CC: Ammar Faizi > CC: "GNU/Weeb Mailing List" > TO: Jiaqi Yan > CC: Andrew Morton > CC: Linux Memory Management List > > tree: https://github.com/ammarfaizi2/linux-block akpm/mm/mm-unstable > head: 57832af0c4af4dc0da3f24703ebefc0ef83b0e1b > commit: 180b0b616c523c240a8635478e563cdfccf493bb [137/175] mm/khugepaged: recover from poisoned file-backed memory > :::::: branch date: 2 days ago > :::::: commit date: 2 days ago > config: x86_64-randconfig-c002 > compiler: gcc-11 (Debian 11.3.0-8) 11.3.0 > > If you fix the issue, kindly add following tag where applicable > | Reported-by: kernel test robot > | Reported-by: Julia Lawall > > cocci warnings: (new ones prefixed by >>) > >> mm/khugepaged.c:2038:27-31: ERROR: invalid reference to the index variable of the iterator on line 2026 > mm/khugepaged.c:2096:21-25: ERROR: invalid reference to the index variable of the iterator on line 2026 > > vim +2038 mm/khugepaged.c > > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1737 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1738 /** > 99cb0dbd47a15d Song Liu 2019-09-23 1739 * collapse_file - collapse filemap/tmpfs/shmem pages into huge one. > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1740 * > 336e6b53d99ae3 Alex Shi 2020-12-14 1741 * @mm: process address space where collapse happens > 34488399fa08fa Zach O'Keefe 2022-09-22 1742 * @addr: virtual collapse start address > 336e6b53d99ae3 Alex Shi 2020-12-14 1743 * @file: file that collapse on > 336e6b53d99ae3 Alex Shi 2020-12-14 1744 * @start: collapse start address > 9710a78ab2aed0 Zach O'Keefe 2022-07-06 1745 * @cc: collapse context and scratchpad > 336e6b53d99ae3 Alex Shi 2020-12-14 1746 * > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1747 * Basic scheme is simple, details are more complex: > 87c460a0bded56 Hugh Dickins 2018-11-30 1748 * - allocate and lock a new huge page; > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1749 * - scan page cache replacing old pages with the new one > 99cb0dbd47a15d Song Liu 2019-09-23 1750 * + swap/gup in pages if necessary; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1751 * + fill in gaps; > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1752 * + keep old pages around in case rollback is required; > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1753 * - if replacing succeeds: > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1754 * + copy data over; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1755 * + free old pages; > 87c460a0bded56 Hugh Dickins 2018-11-30 1756 * + unlock huge page; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1757 * - if replacing failed; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1758 * + put all pages back and unfreeze them; > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1759 * + restore gaps in the page cache; > 87c460a0bded56 Hugh Dickins 2018-11-30 1760 * + unlock and free huge page; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1761 */ > 34488399fa08fa Zach O'Keefe 2022-09-22 1762 static int collapse_file(struct mm_struct *mm, unsigned long addr, > 579c571e2efdb8 Song Liu 2019-09-23 1763 struct file *file, pgoff_t start, > 34488399fa08fa Zach O'Keefe 2022-09-22 1764 struct collapse_control *cc) > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1765 { > 579c571e2efdb8 Song Liu 2019-09-23 1766 struct address_space *mapping = file->f_mapping; > 180b0b616c523c Jiaqi Yan 2022-11-06 1767 struct page *hpage, *page, *tmp; > 95c66f836a4276 Gautam Menghani 2022-10-26 1768 pgoff_t index = 0, end = start + HPAGE_PMD_NR; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1769 LIST_HEAD(pagelist); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1770 XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1771 int nr_none = 0, result = SCAN_SUCCEED; > 99cb0dbd47a15d Song Liu 2019-09-23 1772 bool is_shmem = shmem_file(file); > 95c66f836a4276 Gautam Menghani 2022-10-26 1773 int nr = 0; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1774 > 99cb0dbd47a15d Song Liu 2019-09-23 1775 VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1776 VM_BUG_ON(start & (HPAGE_PMD_NR - 1)); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1777 > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1778 result = alloc_charge_hpage(&hpage, mm, cc); > 9710a78ab2aed0 Zach O'Keefe 2022-07-06 1779 if (result != SCAN_SUCCEED) > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1780 goto out; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1781 > 6b24ca4a1a8d4e Matthew Wilcox (Oracle 2020-06-27 1782) /* > 6b24ca4a1a8d4e Matthew Wilcox (Oracle 2020-06-27 1783) * Ensure we have slots for all the pages in the range. This is > 6b24ca4a1a8d4e Matthew Wilcox (Oracle 2020-06-27 1784) * almost certainly a no-op because most of the pages must be present > 6b24ca4a1a8d4e Matthew Wilcox (Oracle 2020-06-27 1785) */ > 95feeabb77149f Hugh Dickins 2018-11-30 1786 do { > 95feeabb77149f Hugh Dickins 2018-11-30 1787 xas_lock_irq(&xas); > 95feeabb77149f Hugh Dickins 2018-11-30 1788 xas_create_range(&xas); > 95feeabb77149f Hugh Dickins 2018-11-30 1789 if (!xas_error(&xas)) > 95feeabb77149f Hugh Dickins 2018-11-30 1790 break; > 95feeabb77149f Hugh Dickins 2018-11-30 1791 xas_unlock_irq(&xas); > 95feeabb77149f Hugh Dickins 2018-11-30 1792 if (!xas_nomem(&xas, GFP_KERNEL)) { > 95feeabb77149f Hugh Dickins 2018-11-30 1793 result = SCAN_FAIL; > 95feeabb77149f Hugh Dickins 2018-11-30 1794 goto out; > 95feeabb77149f Hugh Dickins 2018-11-30 1795 } > 95feeabb77149f Hugh Dickins 2018-11-30 1796 } while (1); > 95feeabb77149f Hugh Dickins 2018-11-30 1797 > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1798 __SetPageLocked(hpage); > 99cb0dbd47a15d Song Liu 2019-09-23 1799 if (is_shmem) > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1800 __SetPageSwapBacked(hpage); > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1801 hpage->index = start; > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1802 hpage->mapping = mapping; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1803 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1804 /* > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1805 * At this point the hpage is locked and not up-to-date. > 87c460a0bded56 Hugh Dickins 2018-11-30 1806 * It's safe to insert it into the page cache, because nobody would > 87c460a0bded56 Hugh Dickins 2018-11-30 1807 * be able to map it or use it in another way until we unlock it. > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1808 */ > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1809 > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1810 xas_set(&xas, start); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1811 for (index = start; index < end; index++) { > 180b0b616c523c Jiaqi Yan 2022-11-06 1812 page = xas_next(&xas); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1813 > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1814 VM_BUG_ON(index != xas.xa_index); > 99cb0dbd47a15d Song Liu 2019-09-23 1815 if (is_shmem) { > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1816 if (!page) { > 701270fa193aad Hugh Dickins 2018-11-30 1817 /* > 99cb0dbd47a15d Song Liu 2019-09-23 1818 * Stop if extent has been truncated or > 99cb0dbd47a15d Song Liu 2019-09-23 1819 * hole-punched, and is now completely > 99cb0dbd47a15d Song Liu 2019-09-23 1820 * empty. > 701270fa193aad Hugh Dickins 2018-11-30 1821 */ > 701270fa193aad Hugh Dickins 2018-11-30 1822 if (index == start) { > 701270fa193aad Hugh Dickins 2018-11-30 1823 if (!xas_next_entry(&xas, end - 1)) { > 701270fa193aad Hugh Dickins 2018-11-30 1824 result = SCAN_TRUNCATED; > 042a30824871fa Hugh Dickins 2018-11-30 1825 goto xa_locked; > 701270fa193aad Hugh Dickins 2018-11-30 1826 } > 701270fa193aad Hugh Dickins 2018-11-30 1827 xas_set(&xas, index); > 701270fa193aad Hugh Dickins 2018-11-30 1828 } > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1829 if (!shmem_charge(mapping->host, 1)) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1830 result = SCAN_FAIL; > 042a30824871fa Hugh Dickins 2018-11-30 1831 goto xa_locked; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1832 } > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1833 xas_store(&xas, hpage); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1834 nr_none++; > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1835 continue; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1836 } > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1837 > 3159f943aafdba Matthew Wilcox 2017-11-03 1838 if (xa_is_value(page) || !PageUptodate(page)) { > 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1839) struct folio *folio; > 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1840) > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1841 xas_unlock_irq(&xas); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1842 /* swap in or instantiate fallocated page */ > 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1843) if (shmem_get_folio(mapping->host, index, > 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1844) &folio, SGP_NOALLOC)) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1845 result = SCAN_FAIL; > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1846 goto xa_unlocked; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1847 } > 7459c149ae9ca7 Matthew Wilcox (Oracle 2022-09-02 1848) page = folio_file_page(folio, index); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1849 } else if (trylock_page(page)) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1850 get_page(page); > 042a30824871fa Hugh Dickins 2018-11-30 1851 xas_unlock_irq(&xas); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1852 } else { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1853 result = SCAN_PAGE_LOCK; > 042a30824871fa Hugh Dickins 2018-11-30 1854 goto xa_locked; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1855 } > 99cb0dbd47a15d Song Liu 2019-09-23 1856 } else { /* !is_shmem */ > 99cb0dbd47a15d Song Liu 2019-09-23 1857 if (!page || xa_is_value(page)) { > 99cb0dbd47a15d Song Liu 2019-09-23 1858 xas_unlock_irq(&xas); > 99cb0dbd47a15d Song Liu 2019-09-23 1859 page_cache_sync_readahead(mapping, &file->f_ra, > 99cb0dbd47a15d Song Liu 2019-09-23 1860 file, index, > e5a59d308f52bb David Howells 2020-09-04 1861 end - index); > 99cb0dbd47a15d Song Liu 2019-09-23 1862 /* drain pagevecs to help isolate_lru_page() */ > 99cb0dbd47a15d Song Liu 2019-09-23 1863 lru_add_drain(); > 99cb0dbd47a15d Song Liu 2019-09-23 1864 page = find_lock_page(mapping, index); > 99cb0dbd47a15d Song Liu 2019-09-23 1865 if (unlikely(page == NULL)) { > 99cb0dbd47a15d Song Liu 2019-09-23 1866 result = SCAN_FAIL; > 99cb0dbd47a15d Song Liu 2019-09-23 1867 goto xa_unlocked; > 99cb0dbd47a15d Song Liu 2019-09-23 1868 } > 75f360696ce9d8 Song Liu 2019-11-30 1869 } else if (PageDirty(page)) { > 75f360696ce9d8 Song Liu 2019-11-30 1870 /* > 75f360696ce9d8 Song Liu 2019-11-30 1871 * khugepaged only works on read-only fd, > 75f360696ce9d8 Song Liu 2019-11-30 1872 * so this page is dirty because it hasn't > 75f360696ce9d8 Song Liu 2019-11-30 1873 * been flushed since first write. There > 75f360696ce9d8 Song Liu 2019-11-30 1874 * won't be new dirty pages. > 75f360696ce9d8 Song Liu 2019-11-30 1875 * > 75f360696ce9d8 Song Liu 2019-11-30 1876 * Trigger async flush here and hope the > 75f360696ce9d8 Song Liu 2019-11-30 1877 * writeback is done when khugepaged > 75f360696ce9d8 Song Liu 2019-11-30 1878 * revisits this page. > 75f360696ce9d8 Song Liu 2019-11-30 1879 * > 75f360696ce9d8 Song Liu 2019-11-30 1880 * This is a one-off situation. We are not > 75f360696ce9d8 Song Liu 2019-11-30 1881 * forcing writeback in loop. > 75f360696ce9d8 Song Liu 2019-11-30 1882 */ > 75f360696ce9d8 Song Liu 2019-11-30 1883 xas_unlock_irq(&xas); > 75f360696ce9d8 Song Liu 2019-11-30 1884 filemap_flush(mapping); > 75f360696ce9d8 Song Liu 2019-11-30 1885 result = SCAN_FAIL; > 75f360696ce9d8 Song Liu 2019-11-30 1886 goto xa_unlocked; > 74c42e1baacf20 Rongwei Wang 2021-10-28 1887 } else if (PageWriteback(page)) { > 74c42e1baacf20 Rongwei Wang 2021-10-28 1888 xas_unlock_irq(&xas); > 74c42e1baacf20 Rongwei Wang 2021-10-28 1889 result = SCAN_FAIL; > 74c42e1baacf20 Rongwei Wang 2021-10-28 1890 goto xa_unlocked; > 99cb0dbd47a15d Song Liu 2019-09-23 1891 } else if (trylock_page(page)) { > 99cb0dbd47a15d Song Liu 2019-09-23 1892 get_page(page); > 99cb0dbd47a15d Song Liu 2019-09-23 1893 xas_unlock_irq(&xas); > 99cb0dbd47a15d Song Liu 2019-09-23 1894 } else { > 99cb0dbd47a15d Song Liu 2019-09-23 1895 result = SCAN_PAGE_LOCK; > 99cb0dbd47a15d Song Liu 2019-09-23 1896 goto xa_locked; > 99cb0dbd47a15d Song Liu 2019-09-23 1897 } > 99cb0dbd47a15d Song Liu 2019-09-23 1898 } > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1899 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1900 /* > b93b016313b3ba Matthew Wilcox 2018-04-10 1901 * The page must be locked, so we can drop the i_pages lock > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1902 * without racing with truncate. > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1903 */ > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1904 VM_BUG_ON_PAGE(!PageLocked(page), page); > 4655e5e5f38726 Song Liu 2019-11-15 1905 > 4655e5e5f38726 Song Liu 2019-11-15 1906 /* make sure the page is up to date */ > 4655e5e5f38726 Song Liu 2019-11-15 1907 if (unlikely(!PageUptodate(page))) { > 4655e5e5f38726 Song Liu 2019-11-15 1908 result = SCAN_FAIL; > 4655e5e5f38726 Song Liu 2019-11-15 1909 goto out_unlock; > 4655e5e5f38726 Song Liu 2019-11-15 1910 } > 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1911 > 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1912 /* > 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1913 * If file was truncated then extended, or hole-punched, before > 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1914 * we locked the first page, then a THP might be there already. > 58ac9a8993a13e Zach O'Keefe 2022-09-22 1915 * This will be discovered on the first iteration. > 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1916 */ > 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1917 if (PageTransCompound(page)) { > 58ac9a8993a13e Zach O'Keefe 2022-09-22 1918 struct page *head = compound_head(page); > 58ac9a8993a13e Zach O'Keefe 2022-09-22 1919 > 58ac9a8993a13e Zach O'Keefe 2022-09-22 1920 result = compound_order(head) == HPAGE_PMD_ORDER && > 58ac9a8993a13e Zach O'Keefe 2022-09-22 1921 head->index == start > 58ac9a8993a13e Zach O'Keefe 2022-09-22 1922 /* Maybe PMD-mapped */ > 58ac9a8993a13e Zach O'Keefe 2022-09-22 1923 ? SCAN_PTE_MAPPED_HUGEPAGE > 58ac9a8993a13e Zach O'Keefe 2022-09-22 1924 : SCAN_PAGE_COMPOUND; > 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1925 goto out_unlock; > 06a5e1268a5fb9 Hugh Dickins 2018-11-30 1926 } > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1927 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1928 if (page_mapping(page) != mapping) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1929 result = SCAN_TRUNCATED; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1930 goto out_unlock; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1931 } > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1932 > 74c42e1baacf20 Rongwei Wang 2021-10-28 1933 if (!is_shmem && (PageDirty(page) || > 74c42e1baacf20 Rongwei Wang 2021-10-28 1934 PageWriteback(page))) { > 4655e5e5f38726 Song Liu 2019-11-15 1935 /* > 4655e5e5f38726 Song Liu 2019-11-15 1936 * khugepaged only works on read-only fd, so this > 4655e5e5f38726 Song Liu 2019-11-15 1937 * page is dirty because it hasn't been flushed > 4655e5e5f38726 Song Liu 2019-11-15 1938 * since first write. > 4655e5e5f38726 Song Liu 2019-11-15 1939 */ > 4655e5e5f38726 Song Liu 2019-11-15 1940 result = SCAN_FAIL; > 4655e5e5f38726 Song Liu 2019-11-15 1941 goto out_unlock; > 4655e5e5f38726 Song Liu 2019-11-15 1942 } > 4655e5e5f38726 Song Liu 2019-11-15 1943 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1944 if (isolate_lru_page(page)) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1945 result = SCAN_DEL_PAGE_LRU; > 042a30824871fa Hugh Dickins 2018-11-30 1946 goto out_unlock; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1947 } > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1948 > 99cb0dbd47a15d Song Liu 2019-09-23 1949 if (page_has_private(page) && > 99cb0dbd47a15d Song Liu 2019-09-23 1950 !try_to_release_page(page, GFP_KERNEL)) { > 99cb0dbd47a15d Song Liu 2019-09-23 1951 result = SCAN_PAGE_HAS_PRIVATE; > 2f33a706027c94 Hugh Dickins 2020-05-27 1952 putback_lru_page(page); > 99cb0dbd47a15d Song Liu 2019-09-23 1953 goto out_unlock; > 99cb0dbd47a15d Song Liu 2019-09-23 1954 } > 99cb0dbd47a15d Song Liu 2019-09-23 1955 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1956 if (page_mapped(page)) > 869f7ee6f64773 Matthew Wilcox (Oracle 2022-02-15 1957) try_to_unmap(page_folio(page), > 869f7ee6f64773 Matthew Wilcox (Oracle 2022-02-15 1958) TTU_IGNORE_MLOCK | TTU_BATCH_FLUSH); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1959 > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1960 xas_lock_irq(&xas); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1961 xas_set(&xas, index); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1962 > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1963 VM_BUG_ON_PAGE(page != xas_load(&xas), page); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1964 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1965 /* > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1966 * The page is expected to have page_count() == 3: > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1967 * - we hold a pin on it; > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 1968 * - one reference from page cache; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1969 * - one from isolate_lru_page; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1970 */ > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1971 if (!page_ref_freeze(page, 3)) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1972 result = SCAN_PAGE_COUNT; > 042a30824871fa Hugh Dickins 2018-11-30 1973 xas_unlock_irq(&xas); > 042a30824871fa Hugh Dickins 2018-11-30 1974 putback_lru_page(page); > 042a30824871fa Hugh Dickins 2018-11-30 1975 goto out_unlock; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1976 } > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1977 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1978 /* > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1979 * Add the page to the list to be able to undo the collapse if > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1980 * something go wrong. > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1981 */ > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1982 list_add_tail(&page->lru, &pagelist); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1983 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1984 /* Finally, replace with the new page. */ > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1985 xas_store(&xas, hpage); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1986 continue; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1987 out_unlock: > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1988 unlock_page(page); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1989 put_page(page); > 042a30824871fa Hugh Dickins 2018-11-30 1990 goto xa_unlocked; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1991 } > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 1992 nr = thp_nr_pages(hpage); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 1993 > 180b0b616c523c Jiaqi Yan 2022-11-06 1994 if (!is_shmem) { > 09d91cda0e8207 Song Liu 2019-09-23 1995 filemap_nr_thps_inc(mapping); > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 1996 /* > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 1997 * Paired with smp_mb() in do_dentry_open() to ensure > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 1998 * i_writecount is up to date and the update to nr_thps is > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 1999 * visible. Ensures the page cache will be truncated if the > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2000 * file is opened writable. > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2001 */ > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2002 smp_mb(); > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2003 if (inode_is_open_for_write(mapping->host)) { > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2004 result = SCAN_FAIL; > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2005 filemap_nr_thps_dec(mapping); > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2006 goto xa_locked; > eb6ecbed0aa273 Collin Fijalkovich 2021-06-30 2007 } > 09d91cda0e8207 Song Liu 2019-09-23 2008 } > 042a30824871fa Hugh Dickins 2018-11-30 2009 xa_locked: > 042a30824871fa Hugh Dickins 2018-11-30 2010 xas_unlock_irq(&xas); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2011 xa_unlocked: > 042a30824871fa Hugh Dickins 2018-11-30 2012 > 6d9df8a5889c56 Hugh Dickins 2022-02-14 2013 /* > 6d9df8a5889c56 Hugh Dickins 2022-02-14 2014 * If collapse is successful, flush must be done now before copying. > 6d9df8a5889c56 Hugh Dickins 2022-02-14 2015 * If collapse is unsuccessful, does flush actually need to be done? > 6d9df8a5889c56 Hugh Dickins 2022-02-14 2016 * Do it anyway, to clear the state. > 6d9df8a5889c56 Hugh Dickins 2022-02-14 2017 */ > 6d9df8a5889c56 Hugh Dickins 2022-02-14 2018 try_to_unmap_flush(); > 6d9df8a5889c56 Hugh Dickins 2022-02-14 2019 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2020 if (result == SCAN_SUCCEED) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2021 /* > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2022 * Replacing old pages with new one has succeeded, now we > 180b0b616c523c Jiaqi Yan 2022-11-06 2023 * attempt to copy the contents. > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2024 */ > 2af8ff291848cc Hugh Dickins 2018-11-30 2025 index = start; > 180b0b616c523c Jiaqi Yan 2022-11-06 @2026 list_for_each_entry(page, &pagelist, lru) { > 2af8ff291848cc Hugh Dickins 2018-11-30 2027 while (index < page->index) { > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2028 clear_highpage(hpage + (index % HPAGE_PMD_NR)); > 2af8ff291848cc Hugh Dickins 2018-11-30 2029 index++; > 2af8ff291848cc Hugh Dickins 2018-11-30 2030 } > 180b0b616c523c Jiaqi Yan 2022-11-06 2031 if (copy_highpage_mc(hpage + (page->index % HPAGE_PMD_NR), page)) { > 180b0b616c523c Jiaqi Yan 2022-11-06 2032 result = SCAN_COPY_MC; > 180b0b616c523c Jiaqi Yan 2022-11-06 2033 break; > 180b0b616c523c Jiaqi Yan 2022-11-06 2034 } > 180b0b616c523c Jiaqi Yan 2022-11-06 2035 index++; > 180b0b616c523c Jiaqi Yan 2022-11-06 2036 } > 180b0b616c523c Jiaqi Yan 2022-11-06 2037 while (result == SCAN_SUCCEED && index < end) { > 180b0b616c523c Jiaqi Yan 2022-11-06 @2038 clear_highpage(hpage + (page->index % HPAGE_PMD_NR)); > 180b0b616c523c Jiaqi Yan 2022-11-06 2039 index++; > 180b0b616c523c Jiaqi Yan 2022-11-06 2040 } > 180b0b616c523c Jiaqi Yan 2022-11-06 2041 } > 180b0b616c523c Jiaqi Yan 2022-11-06 2042 > 180b0b616c523c Jiaqi Yan 2022-11-06 2043 if (result == SCAN_SUCCEED) { > 180b0b616c523c Jiaqi Yan 2022-11-06 2044 /* > 180b0b616c523c Jiaqi Yan 2022-11-06 2045 * Copying old pages to huge one has succeeded, now we > 180b0b616c523c Jiaqi Yan 2022-11-06 2046 * need to free the old pages. > 180b0b616c523c Jiaqi Yan 2022-11-06 2047 */ > 180b0b616c523c Jiaqi Yan 2022-11-06 2048 list_for_each_entry_safe(page, tmp, &pagelist, lru) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2049 list_del(&page->lru); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2050 page->mapping = NULL; > 042a30824871fa Hugh Dickins 2018-11-30 2051 page_ref_unfreeze(page, 1); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2052 ClearPageActive(page); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2053 ClearPageUnevictable(page); > 042a30824871fa Hugh Dickins 2018-11-30 2054 unlock_page(page); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2055 put_page(page); > 2af8ff291848cc Hugh Dickins 2018-11-30 2056 } > 180b0b616c523c Jiaqi Yan 2022-11-06 2057 > 180b0b616c523c Jiaqi Yan 2022-11-06 2058 xas_lock_irq(&xas); > 180b0b616c523c Jiaqi Yan 2022-11-06 2059 if (is_shmem) > 180b0b616c523c Jiaqi Yan 2022-11-06 2060 __mod_lruvec_page_state(hpage, NR_SHMEM_THPS, nr); > 180b0b616c523c Jiaqi Yan 2022-11-06 2061 else > 180b0b616c523c Jiaqi Yan 2022-11-06 2062 __mod_lruvec_page_state(hpage, NR_FILE_THPS, nr); > 180b0b616c523c Jiaqi Yan 2022-11-06 2063 > 180b0b616c523c Jiaqi Yan 2022-11-06 2064 if (nr_none) { > 180b0b616c523c Jiaqi Yan 2022-11-06 2065 __mod_lruvec_page_state(hpage, NR_FILE_PAGES, nr_none); > 180b0b616c523c Jiaqi Yan 2022-11-06 2066 /* nr_none is always 0 for non-shmem. */ > 180b0b616c523c Jiaqi Yan 2022-11-06 2067 __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2068 } > 180b0b616c523c Jiaqi Yan 2022-11-06 2069 /* Join all the small entries into a single multi-index entry. */ > 180b0b616c523c Jiaqi Yan 2022-11-06 2070 xas_set_order(&xas, start, HPAGE_PMD_ORDER); > 180b0b616c523c Jiaqi Yan 2022-11-06 2071 xas_store(&xas, hpage); > 180b0b616c523c Jiaqi Yan 2022-11-06 2072 xas_unlock_irq(&xas); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2073 > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2074 SetPageUptodate(hpage); > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2075 page_ref_add(hpage, HPAGE_PMD_NR - 1); > 6058eaec816f29 Johannes Weiner 2020-06-03 2076 if (is_shmem) > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2077 set_page_dirty(hpage); > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2078 lru_cache_add(hpage); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2079 > 042a30824871fa Hugh Dickins 2018-11-30 2080 /* > 042a30824871fa Hugh Dickins 2018-11-30 2081 * Remove pte page tables, so we can re-fault the page as huge. > 042a30824871fa Hugh Dickins 2018-11-30 2082 */ > 34488399fa08fa Zach O'Keefe 2022-09-22 2083 result = retract_page_tables(mapping, start, mm, addr, hpage, > 34488399fa08fa Zach O'Keefe 2022-09-22 2084 cc); > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2085 unlock_page(hpage); > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2086 hpage = NULL; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2087 } else { > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2088 /* Something went wrong: roll back page cache changes */ > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2089 xas_lock_irq(&xas); > 2f55f070e5b80f Miaohe Lin 2022-06-25 2090 if (nr_none) { > aaa52e340073b7 Hugh Dickins 2018-11-30 2091 mapping->nrpages -= nr_none; > aaa52e340073b7 Hugh Dickins 2018-11-30 2092 shmem_uncharge(mapping->host, nr_none); > 2f55f070e5b80f Miaohe Lin 2022-06-25 2093 } > aaa52e340073b7 Hugh Dickins 2018-11-30 2094 > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2095 xas_set(&xas, start); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2096 xas_for_each(&xas, page, end - 1) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2097 page = list_first_entry_or_null(&pagelist, > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2098 struct page, lru); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2099 if (!page || xas.xa_index < page->index) { > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2100 if (!nr_none) > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2101 break; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2102 nr_none--; > 59749e6ce53735 Johannes Weiner 2016-12-12 2103 /* Put holes back where they were */ > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2104 xas_store(&xas, NULL); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2105 continue; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2106 } > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2107 > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2108 VM_BUG_ON_PAGE(page->index != xas.xa_index, page); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2109 > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2110 /* Unfreeze the page. */ > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2111 list_del(&page->lru); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2112 page_ref_unfreeze(page, 2); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2113 xas_store(&xas, page); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2114 xas_pause(&xas); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2115 xas_unlock_irq(&xas); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2116 unlock_page(page); > 042a30824871fa Hugh Dickins 2018-11-30 2117 putback_lru_page(page); > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2118 xas_lock_irq(&xas); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2119 } > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2120 VM_BUG_ON(nr_none); > 180b0b616c523c Jiaqi Yan 2022-11-06 2121 /* > 180b0b616c523c Jiaqi Yan 2022-11-06 2122 * Undo the updates of filemap_nr_thps_inc for non-SHMEM file only. > 180b0b616c523c Jiaqi Yan 2022-11-06 2123 * This undo is not needed unless failure is due to SCAN_COPY_MC. > 180b0b616c523c Jiaqi Yan 2022-11-06 2124 */ > 180b0b616c523c Jiaqi Yan 2022-11-06 2125 if (!is_shmem && result == SCAN_COPY_MC) > 180b0b616c523c Jiaqi Yan 2022-11-06 2126 filemap_nr_thps_dec(mapping); > 180b0b616c523c Jiaqi Yan 2022-11-06 2127 > 77da9389b9d5f0 Matthew Wilcox 2017-12-04 2128 xas_unlock_irq(&xas); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2129 > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2130 hpage->mapping = NULL; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2131 } > 042a30824871fa Hugh Dickins 2018-11-30 2132 > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2133 if (hpage) > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2134 unlock_page(hpage); > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2135 out: > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2136 VM_BUG_ON(!list_empty(&pagelist)); > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2137 if (hpage) { > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2138 mem_cgroup_uncharge(page_folio(hpage)); > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2139 put_page(hpage); > c6a7f445a2727a Yang Shi 2022-07-06 2140 } > 95c66f836a4276 Gautam Menghani 2022-10-26 2141 > 95c66f836a4276 Gautam Menghani 2022-10-26 2142 trace_mm_khugepaged_collapse_file(mm, hpage, index, is_shmem, addr, file, nr, result); > 50ad2f24b3b48c Zach O'Keefe 2022-07-06 2143 return result; > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2144 } > f3f0e1d2150b2b Kirill A. Shutemov 2016-07-26 2145 > > -- > 0-DAY CI Kernel Test Service > https://01.org/lkp