Re: [PATCH v2] mm/hugetlb: Defer freeing of huge pages if in non-task context

[Waiman Long <longman@redhat.com>]

On 12/17/19 9:13 AM, Kirill Tkhai wrote:
> On 17.12.2019 17:00, Waiman Long wrote:
>> On 12/17/19 5:50 AM, Kirill Tkhai wrote:
>>> On 17.12.2019 12:31, Michal Hocko wrote:
>>>> On Mon 16-12-19 20:25:08, Waiman Long wrote:
>>>> [...]
>>>>> Both the hugetbl_lock and the subpool lock can be acquired in
>>>>> free_huge_page(). One way to solve the problem is to make both locks
>>>>> irq-safe.
>>>> Please document why we do not take this, quite natural path and instead
>>>> we have to come up with an elaborate way instead. I believe the primary
>>>> motivation is that some operations under those locks are quite
>>>> expensive. Please add that to the changelog and ideally to the code as
>>>> well. We probably want to fix those anyway and then this would be a
>>>> temporary workaround.
>>>>
>>>>> Another alternative is to defer the freeing to a workqueue job.
>>>>>
>>>>> This patch implements the deferred freeing by adding a
>>>>> free_hpage_workfn() work function to do the actual freeing. The
>>>>> free_huge_page() call in a non-task context saves the page to be freed
>>>>> in the hpage_freelist linked list in a lockless manner.
>>>> Do we need to over complicate this (presumably) rare event by a lockless
>>>> algorithm? Why cannot we use a dedicated spin lock for for the linked
>>>> list manipulation? This should be really a trivial code without an
>>>> additional burden of all the lockless subtleties.
>>> Why not llist_add()/llist_del_all() ?
>>>
>> The llist_add() and llist_del_all() are just simple helpers. Because
>> this lockless case involve synchronization of two variables, the llist
>> helpers do not directly apply here. So the rests cannot be used. It will
>> look awkward it is partially converted to use the helpers. If we convert
>> to use a lock as suggested by Michal, using the helpers will be an
>> overkill as xchg() will not be needed.
> I don't understand you. What are two variables?
>
> Why can't you simply do the below?
>
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index ac65bb5e38ac..e8ec753f3d92 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -1136,7 +1136,7 @@ static inline void ClearPageHugeTemporary(struct page *page)
>  	page[2].mapping = NULL;
>  }
>  
> -void free_huge_page(struct page *page)
> +static void __free_huge_page(struct page *page)
>  {
>  	/*
>  	 * Can't pass hstate in here because it is called from the
> @@ -1199,6 +1199,35 @@ void free_huge_page(struct page *page)
>  	spin_unlock(&hugetlb_lock);
>  }
>  
> +static struct llist_head hpage_freelist = LLIST_HEAD_INIT;
> +
> +static void free_hpage_workfn(struct work_struct *work)
> +{
> +	struct llist_node *node;
> +	struct page *page;
> +
> +	node = llist_del_all(&hpage_freelist);
> +
> +	while (node) {
> +		page = container_of(node, struct page, mapping);
> +		node = node->next;
> +		__free_huge_page(page);
> +	}
> +}
> +
> +static DECLARE_WORK(free_hpage_work, free_hpage_workfn);
> +
> +void free_huge_page(struct page *page)
> +{
> +	if (!in_task()) {
> +		if (llist_add((struct llist_node *)&page->mapping, &hpage_freelist))
> +			schedule_work(&free_hpage_work);
> +		return;
> +	}
> +
> +	__free_huge_page(page);
> +}
> +
>  static void prep_new_huge_page(struct hstate *h, struct page *page, int nid)
>  {
>  	INIT_LIST_HEAD(&page->lru);
>
You are right. That should work. I was not aware of the llist before so
I haven't fully grasped its capability. Thanks for the suggestion.

Cheers,
Longman