From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com [209.85.213.172]) by kanga.kvack.org (Postfix) with ESMTP id 59E2D6B0032 for ; Thu, 23 Apr 2015 11:25:31 -0400 (EDT) Received: by igblo3 with SMTP id lo3so26020287igb.1 for ; Thu, 23 Apr 2015 08:25:30 -0700 (PDT) Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com. [2607:f8b0:4001:c05::230]) by mx.google.com with ESMTPS id mn19si4665851icb.96.2015.04.23.08.25.29 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 23 Apr 2015 08:25:29 -0700 (PDT) Received: by igbhj9 with SMTP id hj9so26867303igb.1 for ; Thu, 23 Apr 2015 08:25:29 -0700 (PDT) Message-ID: <55390EE1.8020304@gmail.com> Date: Thu, 23 Apr 2015 11:25:21 -0400 From: Austin S Hemmelgarn MIME-Version: 1.0 Subject: Re: Interacting with coherent memory on external devices References: <20150421214445.GA29093@linux.vnet.ibm.com> <20150422000538.GB6046@gmail.com> <20150422131832.GU5561@linux.vnet.ibm.com> <1429756200.4915.19.camel@kernel.crashing.org> In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms010504080506000003020909" Sender: owner-linux-mm@kvack.org List-ID: To: Christoph Lameter , Benjamin Herrenschmidt Cc: "Paul E. McKenney" , Jerome Glisse , linux-kernel@vger.kernel.org, linux-mm@kvack.org, jglisse@redhat.com, mgorman@suse.de, aarcange@redhat.com, riel@redhat.com, airlied@redhat.com, aneesh.kumar@linux.vnet.ibm.com, Cameron Buschardt , Mark Hairgrove , Geoffrey Gerfin , John McKenna , akpm@linux-foundation.org This is a cryptographically signed message in MIME format. --------------ms010504080506000003020909 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable On 2015-04-23 10:25, Christoph Lameter wrote: > On Thu, 23 Apr 2015, Benjamin Herrenschmidt wrote: > >> They are via MMIO space. The big differences here are that via CAPI th= e >> memory can be fully cachable and thus have the same characteristics as= >> normal memory from the processor point of view, and the device shares >> the MMU with the host. >> >> Practically what that means is that the device memory *is* just some >> normal system memory with a larger distance. The NUMA model is an >> excellent representation of it. > > I sure wish you would be working on using these features to increase > performance and the speed of communication to devices. > > Device memory is inherently different from main memory (otherwise the > device would be using main memory) and thus not really NUMA. NUMA at le= ast > assumes that the basic characteristics of memory are the same while jus= t > the access speeds vary. GPU memory has very different performance > characteristics and the various assumptions on memory that the kernel > makes for the regular processors may not hold anymore. > You are restricting your definition of NUMA to what the industry=20 constrains it to mean. Based solely on the academic definition of a=20 NUMA system, this _is_ NUMA. In fact, based on the academic definition, = all modern systems could be considered to be NUMA systems, with each=20 level of cache representing a memory only node. Looking at this whole conversation, all I see is two different views on=20 how to present the asymmetric multiprocessing arrangements that have=20 become commonplace in today's systems to userspace. Your model favors=20 performance, while CAPI favors simplicity for userspace. --------------ms010504080506000003020909 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGuDCC BrQwggScoAMCAQICAxBuVTANBgkqhkiG9w0BAQ0FADB5MRAwDgYDVQQKEwdSb290IENBMR4w HAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmlu ZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZzAeFw0xNTAz MjUxOTM0MzhaFw0xNTA5MjExOTM0MzhaMGMxGDAWBgNVBAMTD0NBY2VydCBXb1QgVXNlcjEj MCEGCSqGSIb3DQEJARYUYWhmZXJyb2luN0BnbWFpbC5jb20xIjAgBgkqhkiG9w0BCQEWE2Fo ZW1tZWxnQG9oaW9ndC5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCdD/zW 2rRAFCLnDfXpWxU1+ODqRVUgzHvrRO7ADUxRo1CBDc3JSX5TIW2OGmQ3DAKGOACp8Z0sgxMc B05tzAZ/M7m4jajVrwwdVCdrwVGxTdAai7Kwg4ZCVfyMVhcwo8R2eW3QahBx34G0RKumK9sZ ZQSQ+zULAzpY6uz7T1sAk/erMoivRXF6u8WvOsLkOD1F/Xyv1ZccSUG5YeDgZgc0nZUBvyIp zXSHjgWerFkrxEM3y2z/Ff3eL1sgGYecV/I1F+I5S01V7Kclt/qRW10c/4JEGRcI1FmrJBPu BtMYPbg/3Y9LZROYN+mVIFxZxOfrmjfFZ96xt/TaMXo8vcEKtWcNEjhGBjEbfMUEm4aq8ygQ 4MuEcpJc8DJCHBkg2KBk13DkbU2qNepTD6Uip1C+g+KMr0nd6KOJqSH27ZuNY4xqV4hIxFHp ex0zY7mq6fV2o6sKBGQzRdI20FDYmNjsLJwjH6qJ8laxFphZnPRpBThmu0AjuBWE72GnI1oA aO+bs92MQGJernt7hByCnDO82W/ykbVz+Ge3Sax8NY0m2Xdvp6WFDY/PjD9CdaJ9nwQGsUSa N54lrZ2qMTeCI9Vauwf6U69BA42xgk65VvxvTNqji+tZ4aZbarZ7el2/QDHOb/rRwlCFplS/ z4l1f1nOrE6bnDl5RBJyW3zi74P6GwIDAQABo4IBWTCCAVUwDAYDVR0TAQH/BAIwADBWBglg hkgBhvhCAQ0ESRZHVG8gZ2V0IHlvdXIgb3duIGNlcnRpZmljYXRlIGZvciBGUkVFIGhlYWQg b3ZlciB0byBodHRwOi8vd3d3LkNBY2VydC5vcmcwDgYDVR0PAQH/BAQDAgOoMEAGA1UdJQQ5 MDcGCCsGAQUFBwMEBggrBgEFBQcDAgYKKwYBBAGCNwoDBAYKKwYBBAGCNwoDAwYJYIZIAYb4 QgQBMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcwAYYWaHR0cDovL29jc3AuY2FjZXJ0Lm9y ZzAxBgNVHR8EKjAoMCagJKAihiBodHRwOi8vY3JsLmNhY2VydC5vcmcvcmV2b2tlLmNybDA0 BgNVHREELTArgRRhaGZlcnJvaW43QGdtYWlsLmNvbYETYWhlbW1lbGdAb2hpb2d0LmNvbTAN BgkqhkiG9w0BAQ0FAAOCAgEAGvl7xb42JMRH5D/vCIDYvFY3dR2FPd5kmOqpKU/fvQ8ovmJa p5N/FDrsCL+YdslxPY+AAn78PYmL5pFHTdRadT++07DPIMtQyy2qd+XRmz6zP8Il7vGcEDmO WmMLYMq4xV9s/N7t7JJp6ftdIYUcoTVChUgilDaRWMLidtslCdRsBVfUjPb1bF5Ua31diKDP e0M9/e2CU36rbcTtiNCXhptMigzuL3zJXUf2B9jyUV8pnqNEQH36fqJ7YTBLcpq3aYa2XbAH Hgx9GehJBIqwspDmhPCFZ/QmqUXCkt+XfvinQ2NzKR6P3+OdYbwqzVX8BdMeojh7Ig8x/nIx mQ+/ufstL1ZYp0bg13fyK/hPYSIBpayaC76vzWovkIm70DIDRIFLi20p/qTd7rfDYy831Hjm +lDdCECF9bIXEWFk33kA97dgQIMbf5chEmlFg8S0e4iw7LMjvRqMX3eCD8GJ2+oqyZUwzZxy S0Mx+rBld5rrN7LsXwZ671HsGqNeYbYeU25e7t7/Gcc6Bd/kPfA+adEuUGFcvUKH3trDYqNq 6mOkAd8WO/mQadlc3ztS++XDMhmIpfBre9MPAr6usqf+wc+R8Nk9KLK39kEgrqVfzc/fgf8L MaD4rHnusdg4gca6Yi+kNrm99anw7SwaBrBvULYBp7ixNRUhaYiNW4YjTrYxggShMIIEnQIB ATCBgDB5MRAwDgYDVQQKEwdSb290IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5v cmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEW EnN1cHBvcnRAY2FjZXJ0Lm9yZwIDEG5VMAkGBSsOAwIaBQCgggH1MBgGCSqGSIb3DQEJAzEL BgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE1MDQyMzE1MjUyMVowIwYJKoZIhvcNAQkE MRYEFHv7W7/H3OoJZEzy3M3wVmPBvhMuMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEq MAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwIC AUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZEGCSsGAQQBgjcQBDGBgzCBgDB5MRAwDgYD VQQKEwdSb290IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMT GUNBIENlcnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2Fj ZXJ0Lm9yZwIDEG5VMIGTBgsqhkiG9w0BCRACCzGBg6CBgDB5MRAwDgYDVQQKEwdSb290IENB MR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2ln bmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZwIDEG5V MA0GCSqGSIb3DQEBAQUABIICAJgmceESJ6F394pUTUc+0x+vMuoZaO+3qYAD90OmwiHUpigz 5oBmA5Fwf+/5Mrv5vm1au80VUp+xj4x9EMsa9d24I9YHtH88iARcswVTD0QQGozmN6e+uSu0 TDAmqTtX5opWPirY4h54RtCY9uqtVBTfADoN/GWyDfdVND61OPWG2N6MvGOP1ecSR6xMy89/ 5Ug+vZMK9HU1Pxc+H+nzkiG3aD9tU9P/PZsw5sborGgaPkf3vQlQ0MNRDlzholiu4B6oIOEn z8NVqWrhC4ldZbd9WR+TYyTSpPyJKMC1gQ4N11qoV+4CuJ6Tc6i+zP3aCq1HSUBatWg4jPq8 AF++hp7bcEGh5TYX1AjBy03aSAywK98haRFoBRzEpdWiXS9exH4MDe2ZZj1bWwNYfCqbwA1G nnvzJZbEcQf9XYPNlDEgVcGHhFCC5u2JGEOYGLiUUoSAJsewbzy5EMgi7GgG8PWCP9pMM6hj SxdpE1TuPpPq8a6Wm89bhcqDq8ACOY/o62MNKEy1Ku+Z2SzKjxSPbW58qmvhWhYIjvIwmBlG EWy8RxBoOhlq2J12t9S6ktUFsEbpeZGqJ6Yv9Vm0S5UEX1S1nL766X6t5cj9bVbm0v6mKj3n SBdY+/y3TzzPzBnJAYKE3euChpD6/kdL+idmmvek6sx4cpz9dVcTRabwsPGWAAAAAAAA --------------ms010504080506000003020909-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org