From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28CFEC54EBD for ; Fri, 13 Jan 2023 08:41:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 875258E0002; Fri, 13 Jan 2023 03:41:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 823AB8E0001; Fri, 13 Jan 2023 03:41:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C4A58E0002; Fri, 13 Jan 2023 03:41:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5C3198E0001 for ; Fri, 13 Jan 2023 03:41:15 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 23A2540262 for ; Fri, 13 Jan 2023 08:41:15 +0000 (UTC) X-FDA: 80349131310.14.0EF68F6 Received: from domac.alu.hr (domac.alu.unizg.hr [161.53.235.3]) by imf25.hostedemail.com (Postfix) with ESMTP id BB1B0A000C for ; Fri, 13 Jan 2023 08:41:11 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=alu.unizg.hr header.s=mail header.b=hYuFzhPh; dkim=pass header.d=alu.unizg.hr header.s=mail header.b=QD56bSbr; spf=pass (imf25.hostedemail.com: domain of mirsad.todorovac@alu.unizg.hr designates 161.53.235.3 as permitted sender) smtp.mailfrom=mirsad.todorovac@alu.unizg.hr; dmarc=pass (policy=none) header.from=alu.unizg.hr ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673599272; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jdRryInB3zPKQDh7tu926RiUdorkcudtEbLeZ8TcXo0=; b=pWFc/JGTm+mt9CQgrNzIeEq+MqPsnoZ+54NDMssdZ5hEvKvG51yOI67kZHcZHoI36vUhwq Cxjv030C3Q1TWBdNZR13VtmDhWYxuyVDcg9hMwJ9rUKCyelQk8zWv7urUzL7aOe94vK0ct IvFmlYB4IcX4fnZ05Hiy+opXKUBh1gE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=alu.unizg.hr header.s=mail header.b=hYuFzhPh; dkim=pass header.d=alu.unizg.hr header.s=mail header.b=QD56bSbr; spf=pass (imf25.hostedemail.com: domain of mirsad.todorovac@alu.unizg.hr designates 161.53.235.3 as permitted sender) smtp.mailfrom=mirsad.todorovac@alu.unizg.hr; dmarc=pass (policy=none) header.from=alu.unizg.hr ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673599272; a=rsa-sha256; cv=none; b=ndoQWIfHIp0avk/5OQsNbIomm2LDwR6QSeI0y4AB1tkOIYuemxJuk4IT7/zWIXrYDIDjE7 v5pnAbfpcBzh4I4Wnb3kXo06hfw+dO9zXHsPOzeBU1Xv3v77rQewOKQ3uQOY158LIDv4v4 ZH2BZ8/6P1XG/XRM/x6bDMzODWcwaRI= Received: from localhost (localhost [127.0.0.1]) by domac.alu.hr (Postfix) with ESMTP id EF77E604FC; Fri, 13 Jan 2023 09:41:08 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=alu.unizg.hr; s=mail; t=1673599268; bh=JtWdty7Ep6O5ZrJGhOIoCVCdUQYdp19mGcP2WzzMLkQ=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=hYuFzhPhnqP+3nznK2/+qE7ow3+MJMcM0SEfGiM8Z6LKN5CE7nS0+DY0/ft8Yj35R OAMp1GrT8kKjvR0nl/j/za+oGuRn46VV1xaSKvjZ4DRjn7PCCgKYhbRRoYhwkASgNR qXkWWCTR6PyaHxET4K3umiCr0zWunzpXgAIv/AmcNB0pwkUihhhdUhZoUC8CM48KQz TIPiYCyqXyB2SRO/fEOtpdIE21o4YntsmWJCUz0e4ie7/tuvylwcWqQrN34rp2T61H Y/cd6ia01LxSxznBA0TvIYlvkQpi4d5k+gP8QGhJLPQflHnMHIp7hayupYQsrY0nSk 77TrJQe+J9f5A== X-Virus-Scanned: Debian amavisd-new at domac.alu.hr Received: from domac.alu.hr ([127.0.0.1]) by localhost (domac.alu.hr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fbC26UwPkoBq; Fri, 13 Jan 2023 09:41:06 +0100 (CET) Received: from [161.53.83.51] (pc-mtodorov.grf.hr [161.53.83.51]) by domac.alu.hr (Postfix) with ESMTPSA id ABF6B604FA; Fri, 13 Jan 2023 09:41:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=alu.unizg.hr; s=mail; t=1673599266; bh=JtWdty7Ep6O5ZrJGhOIoCVCdUQYdp19mGcP2WzzMLkQ=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=QD56bSbraPZrW9tBL67DfRCXbEUTtP4vrleOe1mgBFImOxk6cIULL5vl95IW7jHsX uhtj9IThuhPcogUNvUXOBZpABERP338hwBHgj/dKhxogdsaEfLzTrlxoC3j5URtS2n uQImn5UOMQhBPjGxb1Ua0+efcCqIXQPotVNvdYM+dqph2wUDoJU9soeQBT8KjTlCXl SnBpakOkRTvs01pmoI7kah6MZBWhdNmtcz6Z1ZvHn7KHdqRemDUiq+jHMUxoWGGuZ2 oo5tIlM4hD/UOeXc3hmM3FDEzG/zF70fpho1ug5WkVieyXT9BVClevCAX63ObEyuGr IsinaJUve/vPA== Message-ID: Date: Fri, 13 Jan 2023 09:41:05 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: =?UTF-8?B?UmU6IOetlOWkjTogSU5GTzogW0JJU0VDVEVEXSBSZWdyZXNzaW9uOiBB?= =?UTF-8?Q?_Problem_with_/sys/kernel/debug/kmemleak_output=3a_backtrace_not_?= =?UTF-8?Q?printed_since_6=2e2=2e0-rc1?= Content-Language: en-US To: =?UTF-8?B?6buE5pyd6ZizIChaaGFveWFuZyBIdWFuZyk=?= , "linux-kernel@vger.kernel.org" , =?UTF-8?B?546L56eRIChLZSBXYW5nKQ==?= Cc: Catalin Marinas , Andrew Morton , "linux-mm@kvack.org" , Thorsten Leemhuis References: <5272a819-ef74-65ff-be61-4d2d567337de@alu.unizg.hr> <4c467851-8080-44d3-d017-b0e283896119@alu.unizg.hr> <53c2d558-c6a1-38e5-5739-28fff023558d@alu.unizg.hr> <3ba5f8ee-d9bd-69ae-bb3a-c61e47020e2c@alu.unizg.hr> <314a477e7f0644fbaf8fc607daefe70e@BJMBX01.spreadtrum.com> From: Mirsad Goran Todorovac In-Reply-To: <314a477e7f0644fbaf8fc607daefe70e@BJMBX01.spreadtrum.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: BB1B0A000C X-Stat-Signature: 8k4ch6bh9pnker39aw9rpk5dzaequneo X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673599271-372645 X-HE-Meta: U2FsdGVkX1+4WdFJKwZkWbtLlPgXY8osd7/odsk18WGln05xHRLiqO/+qO5RtycGaH6aoRbl8H0xrLDp4WsBRddnPg8ima2V6YyOGyQPNmX/9nbw4SWs2zYzN1qeb1d0eofXQRqGV6eiqdwTAy8tGY36JAPOupoQ7Ac2za6bqN4dFbOs4SHmmDQD98e9PdoFxGMlJSEXgy920MS9Oeq0mvJHpoBMKj9Y12wy6CbGGIzmakWJYM3o0P45LLlJRDDvWYdZNl5wN7JcCmDeLMDj9bAPuqmVb/2e2wcy+umsUWmhnlUndsvySP2c3ppcYLoc9bcaNd4yKiioNIh+FcBrBC353I5SFYhDgfoKEVou3hd+KrqVir28W/A590lLysAk3CwpC659psIdT5C/UqN2fqNuviLcXnSksGQ3FgVNAznS92ZVZYzGU9Y8n86BJATvUQc1HS8v0OYvjQgR+RZNtkZATBUiS97H/W39V+UU358wyV/Xn73g1u+62KTtBQiyR7QQThJSq6WZKwd6R+9rN+ZyyNhhJh5APX2GoKtyeFudsKFlWI8+ZafG+++IwabmEB7XtRVVlv0Mg6C4W3AswNX9olBzhQdWwHbNaQvtuHsLWBjJDyiIB6371RjvJ/79qlETQvKIUwF6gIQNlPeIlzPtPJt7VAKtMKkwH+zWu9QKjuiD6crn/w/HlaLygvmzfL6JFBcz3uzCxgFjie0Gx08lWZrbxFYGXJss+dezFtQCJu6xKvg8VFgwMVLaZMDw/oletG02I8T8d8OjiXewBEE9IhO83qD6BAs+7RWc73r2TTzimlT7NUDSCacadq05VPEu5gMRbgtDu7IdxQkEagHe/uU+Vt1NDw4khuYzepLSB8fjf2hUvSQ22kMdbtK+ZTp7LKGXSeY+8caATJxxhY20VHMn4Neaq4qSneG8ZhqsKcOHqKzHnA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 13.1.2023. 7:20, 黄朝阳 (Zhaoyang Huang) wrote: >> /sys/kernel/debug/kmemleak output: backtrace not printed since 6.2.0-rc1 >> >> >>> On 12. 01. 2023. 20:50, Mirsad Todorovac wrote: >>>> Hi all, >>>> >>>> there seems to be a problem with the output of >>> /sys/kernel/debug/kmemleak: >>>> >>>> [root@pc-mtodorov ~]# cat /sys/kernel/debug/kmemleak unreferenced >>>> object 0xffff951c118568b0 (size 16): >>>> comm "kworker/u12:2", pid 56, jiffies 4294893952 (age 4356.548s) >>>> hex dump (first 16 bytes): >>>> 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0....... >>>> backtrace: >>>> [root@pc-mtodorov ~]# >>>> >>>> Apparently, backtrace of called functions on the stack is no longer >>>> printed >>> with the list of memory leaks. >>>> >>>> This appeared on Lenovo desktop 10TX000VCR, with AlmaLinux 8.7 and >>>> BIOS version M22KT49A (11/10/2022) and 6.2-rc1 and 6.2-rc2 builds. >>>> >>>> This worked on 6.1 with the same CONFIG_KMEMLEAK=y and MGLRU >>> enabled >>>> on a vanilla mainstream kernel from Mr. Torvalds' tree. I don't know >>>> if this is >>> deliberate feature for some reason or a bug. >>>> >>>> Please find attached the config, lshw and kmemleak output. >>>> >>>> The bisection gave this commit as the culprit for the severed stack >>>> backtrace print in /sys/kernel/debug/kmemleak: >>>> >>>> mtodorov@domac:~/linux/kernel/linux_torvalds$ git bisect good >>>> 56a61617dd2276cbc56a6c868599716386d70041 is the first bad commit >>>> commit 56a61617dd2276cbc56a6c868599716386d70041 >>>> Author: Zhaoyang Huang >>>> Date: Thu Oct 27 17:50:24 2022 +0800 >>>> >>>> mm: use stack_depot for recording kmemleak's backtrace >>>> >>>> Using stack_depot to record kmemleak's backtrace which has been >>>> implemented on slub for reducing redundant information. >>>> >>>> The complete bisect log is: >>>> >>>> mtodorov@domac:~/linux/kernel/linux_torvalds$ git bisect log git >>>> bisect start '--' 'mm' >>>> # good: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1 git >>>> bisect good 830b3c68c1fb1e9176028d02ef86f3cf76aa2476 >>>> # bad: [1b929c02afd37871d5afb9d498426f83432e71c2] Linux 6.2-rc1 git >>>> bisect bad 1b929c02afd37871d5afb9d498426f83432e71c2 >>>> # good: [8b9ed79c2d587bec5f603d66801478a5af9af842] Merge tag >>>> 'x86_asm_for_v6.2' of >>>> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip >>>> git bisect good 8b9ed79c2d587bec5f603d66801478a5af9af842 >>>> # good: [e83b39d6bbdb6d25bd6f5c258832774635d29b47] mm: make >>>> drop_caches keep reclaiming on all nodes git bisect good >>>> e83b39d6bbdb6d25bd6f5c258832774635d29b47 >>>> # good: [9997bc017549acd6425e32300eff28424ffeeb6b] zsmalloc: >>> implement >>>> writeback mechanism for zsmalloc git bisect good >>>> 9997bc017549acd6425e32300eff28424ffeeb6b >>>> # good: [6287b7dae80944bfa37784a8f9d6861a4facaa6e] mm,thp,rmap: fix >>>> races between updates of subpages_mapcount git bisect good >>>> 6287b7dae80944bfa37784a8f9d6861a4facaa6e >>>> # good: [8fa590bf344816c925810331eea8387627bbeb40] Merge tag >>>> 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm >>>> git bisect good 8fa590bf344816c925810331eea8387627bbeb40 >>>> # good: [4f292c4de4f6fb83776c0ff22674121eb6ddfa2f] Merge tag >>>> 'x86_mm_for_6.2_v2' of >>>> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip >>>> git bisect good 4f292c4de4f6fb83776c0ff22674121eb6ddfa2f >>>> # bad: [1ea9d333ba475041efe43d9d9bc32e64aea2ea2b] Merge tag >>>> 'mm-stable-2022-12-17-2' of >>>> git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm >>>> git bisect bad 1ea9d333ba475041efe43d9d9bc32e64aea2ea2b >>>> # bad: [56a61617dd2276cbc56a6c868599716386d70041] mm: use >>> stack_depot >>>> for recording kmemleak's backtrace git bisect bad >>>> 56a61617dd2276cbc56a6c868599716386d70041 >>>> # good: [61b963b52f59524e27692bc1c14bfb2459e32eb3] mm/gup_test: >>> free >>>> memory allocated via kvcalloc() using kvfree() git bisect good >>>> 61b963b52f59524e27692bc1c14bfb2459e32eb3 >>>> # first bad commit: [56a61617dd2276cbc56a6c868599716386d70041] mm: >>> use >>>> stack_depot for recording kmemleak's backtrace # good: >>>> [9102b78b6f6ae6af3557114c265c266b312c1319] maple_tree: update >>>> copyright dates for test code git bisect good >>>> 9102b78b6f6ae6af3557114c265c266b312c1319 >>>> # first bad commit: [56a61617dd2276cbc56a6c868599716386d70041] mm: >>> use >>>> stack_depot for recording kmemleak's backtrace You have mail in >>>> /var/mail/mtodorov mtodorov@domac:~/linux/kernel/linux_torvalds$ >>>> >>>> The platform is 10TX000VCR >>> (LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB) >>>> running AlmaLinux 8.7 (CentOS clone) and BIOS M22KT49A. >>>> >>>> GCC used was: >>>> >>>> mtodorov@domac:~/linux/kernel/linux_torvalds$ gcc --version gcc >>>> (Debian 8.3.0-6) 8.3.0 Copyright (C) 2018 Free Software Foundation, >>>> Inc. >>>> This is free software; see the source for copying conditions. There >>>> is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A >>> PARTICULAR PURPOSE. >>>> mtodorov@domac:~/linux/kernel/linux_torvalds$ >>>> >>>> Hope this helps. >>>> >>>> I lack the insight to pinpoint the exact fault in the patch. >>> >>> P.S. >>> >>> To further complicate things, on Ubuntu 22.10 Lenovo Ideapad 3 15ITL6 >>> laptop the kmemleak stack backtrace is printed correctly with the same >>> kernels (but .deb instead of .rpm for the same build): >>> >>> unreferenced object 0xffff94ca88e042d0 (size 80): >>> comm "thermald", pid 768, jiffies 4294893592 (age 78581.504s) >>> hex dump (first 32 bytes): >>> 00 00 00 00 00 00 00 00 0d 01 2d 00 00 00 00 00 ..........-..... >>> af 07 01 80 fd ac ff ff 00 00 00 00 00 00 00 00 ................ >>> backtrace: >>> [] slab_post_alloc_hook+0x91/0x320 >>> [] kmem_cache_alloc+0x169/0x2f0 >>> [] acpi_os_acquire_object+0x2c/0x32 >>> [] acpi_ps_alloc_op+0x4a/0x99 >>> [] acpi_ps_get_next_arg+0x611/0x761 >>> [] acpi_ps_parse_loop+0x494/0x8d7 >>> [] acpi_ps_parse_aml+0x1bb/0x561 >>> [] acpi_ps_execute_method+0x20f/0x2d5 >>> [] acpi_ns_evaluate+0x34d/0x4f3 >>> [] acpi_evaluate_object+0x180/0x3ae >>> [] acpi_run_osc+0x128/0x250 >>> [] int3400_thermal_run_osc+0x6f/0xc0 >>> [int3400_thermal] >>> [] current_uuid_store+0xe3/0x120 [int3400_thermal] >>> [] dev_attr_store+0x14/0x30 >>> [] sysfs_kf_write+0x38/0x50 >>> [] kernfs_fop_write_iter+0x146/0x1d0 You have new >>> mail in /var/mail/root root@marvin-IdeaPad-3-15ITL6:/home/marvin# >>> >>> Regards, >>> Mirsad >> >> thanks for head up. Could you please help to check If CONFIG_STACKTRACE >> and CONFIG_STACKDEPOT is enabled on the fault kernel? > Please try the patch below if the configs are right. I think it should be caused by the late ready for kmemleak > > commit 556eb1927da99ab366e370d593fc32da946768c3 > Author: Zhaoyang Huang > Date: Fri Jan 13 14:13:39 2023 +0800 > > mm: instroduce a flag to indicate stack_depot's ready > > try to solve the kmemleak's late ready issue. > > diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 762b91f..7eb24bc 100644 > --- a/mm/kmemleak.c > +++ b/mm/kmemleak.c > @@ -216,6 +216,7 @@ struct kmemleak_object { static int kmemleak_warning; > /* set if a fatal kmemleak error has occurred */ static int kmemleak_error; > +static int kmemleak_stadep_initialized; > > /* minimum and maximum address that may be valid pointers */ static unsigned long min_addr = ULONG_MAX; @@ -601,7 +602,7 @@ static noinline depot_stack_handle_t set_track_prepare(void) > unsigned long entries[MAX_TRACE]; > unsigned int nr_entries; > > - if (!kmemleak_initialized) > + if (!kmemleak_stadep_initialized) > return 0; > nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 3); > trace_handle = stack_depot_save(entries, nr_entries, GFP_NOWAIT); @@ -2110,6 +2111,7 @@ void __init kmemleak_init(void) > create_object((unsigned long)__start_ro_after_init, > __end_ro_after_init - __start_ro_after_init, > KMEMLEAK_GREY, GFP_ATOMIC); > + kmemleak_stadep_initialized = 1; > } Hi, Zhaoyang, Sorry, applied the patch and rebuilt, but it didn't work for me: [root@pc-mtodorov marvin]# uname -rms Linux 6.2.0-rc3-mglru-kmemlk-patch-zhaoyang1+ x86_64 [root@pc-mtodorov marvin]# cat /sys/kernel/debug/kmemleak [root@pc-mtodorov marvin]# echo scan | tee !$ echo scan | tee /sys/kernel/debug/kmemleak scan [root@pc-mtodorov marvin]# cat /sys/kernel/debug/kmemleak unreferenced object 0xffff98938fed1ee0 (size 16): comm "kworker/u12:2", pid 57, jiffies 4294894053 (age 66.616s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0....... backtrace: [root@pc-mtodorov marvin]# Still the stack backtrace list is severed. CONFIG_STACKDEPOT and CONFIG_STACKTRACE are both "=y": [root@pc-mtodorov marvin]# grep -E 'CONFIG_(STACKTRACE|STACKDEPOT)' /boot/config-6.2.0-rc3-mglru-kmemlk-patch-zhaoyang1+ CONFIG_STACKTRACE_SUPPORT=y CONFIG_STACKDEPOT=y # CONFIG_STACKTRACE_BUILD_ID is not set CONFIG_STACKTRACE=y [root@pc-mtodorov marvin]# Your patch was successfully applied (after some tweaks): mtodorov@domac:~/linux/kernel/linux_torvalds$ git diff mm diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 92f670edbf51..78fd3b89b057 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -216,6 +216,7 @@ static int kmemleak_initialized; static int kmemleak_warning; /* set if a fatal kmemleak error has occurred */ static int kmemleak_error; +static int kmemleak_stadep_initialized; /* minimum and maximum address that may be valid pointers */ static unsigned long min_addr = ULONG_MAX; @@ -601,7 +602,7 @@ static noinline depot_stack_handle_t set_track_prepare(void) unsigned long entries[MAX_TRACE]; unsigned int nr_entries; - if (!kmemleak_initialized) + if (!kmemleak_stadep_initialized) return 0; nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 3); trace_handle = stack_depot_save(entries, nr_entries, GFP_NOWAIT); @@ -2110,6 +2111,7 @@ void __init kmemleak_init(void) create_object((unsigned long)__start_ro_after_init, __end_ro_after_init - __start_ro_after_init, KMEMLEAK_GREY, GFP_ATOMIC); + kmemleak_stadep_initialized = 1; } /* mtodorov@domac:~/linux/kernel/linux_torvalds$ Please verify that the above is correct. Regards, Mirsad -- Mirsad Todorovac System engineer Faculty of Graphic Arts | Academy of Fine Arts University of Zagreb Republic of Croatia, the European Union -- Sistem inženjer Grafički fakultet | Akademija likovnih umjetnosti Sveučilište u Zagrebu