If you modify gnome, an attacker could create files by calling system calls directly.

You need to modify the kernel if you really want to block any software from creating files with certain characters.

Actually, I think a corollary of my question is if you restrict filenames, then what happens when some program you install does try to create one? (granted, in Linux most packages do not have junk in filenames - I doubt if any do)

So you would want to leave a backdoor so that the system can create them when needed, but userspace is prevented from casually creating litter. Thus, absolute control of filenames is not necessary, but the limitations should be reasonable.

But the approach in this question (eg. modifying Gnome file manager) would still not prevent me from accidentally creating litter with something as simple as a touch/xargs pipeline gone wrong.

Although perhaps SELinux or one of the other advanced security frameworks can block it for the user?

Since you write the code, you can tell your kernel that no program can create such files. Or maybe you can program it so that only root can create such files. Or maybe programs that have some capability (read https://man7.org/linux/man-pages/man7/capabilities.7.html).

Regarding the question of what happens if a program tries to create such a file, expect something like this (assuming you patch the kernel to return -EINVAL as I suggested):

$ errno EINVAL
EINVAL 22 Invalid argument

(The following is made up; I didn't really patch my kernel to do that; yet.)

$ touch 'foo\n'
touch: cannot touch 'foo\n': Invalid argument

The current open(2) manual page already covers the possibility that certain filesystems may restrict the file names:

     EINVAL
            The final component ("basename") of  pathname  is  invalid
            (e.g.,  it contains characters not permitted by the under‐
            lying filesystem).