Is there any way to run unit tests for vault application?

You can run unit tests on your local machine, or on a custom runner that has M-Files installed.  You cannot run unit tests on a standard shared GitHub runner as it does not have M-Files installed.

For example: internally we use GitLab (we use GitHub for public samples, but GitLab internally) for vault application projects, and we use custom runners set up that have M-Files installed.  These runners are used to execute both unit and integration tests as part of CI/CD processes.

We don't have any public guidance for this, but creating custom runners is something that someone with some experience of your git environment and CI processes should be able to guide you on.

What we were trying to do is that - downloading the installer and then install it in the GitHub runner. Although we are still not being able to resolve the issue. 

I have not tried this approach with GitHub so cannot comment further, unfortunately.