Technically you could write an event handler (BeforeCreateNewObjectFinalize) to get and check the extension of the files being uploaded, and throw an exception error where needed. Although in practice I'm not sure how much of an overhead would this be for the vault in terms of performance. This needs to be tested as it needs to run for every single document creation in the vault.
If you're interested, here you can find the relevant M-Files API method that can be used for that purpose:
