probably you have 2FA activated, and this way no regular login with the UI password is possible outside of mailcow UI, because you would not be able to provide the second factor via Outlook to mailcow server.
You have to use app passwords, which every user can create in his mailcow UI, see the yellow box:
https://docs.mailcow.email/manual-guides/mailcow-UI/u_e-mailcow_ui-tfa/
Same applies also if you use SSO/OIDC:
https://docs.mailcow.email/manual-guides/mailcow-UI/u_e-mailcow_ui-generic-oidc/#authentication-for-external-mail-clients-imap-sieve-pop3-smtp
The problem is authentication goes through mailcow only. No chance of getting a custom popup so far. If this is possible at all I cannot tell, you could phrase a feature request on github.
Switching to a different client like Thunderbird would also make first time client configuration easier at it pulls all IMAP CAldav etc. information from DNS. You just provide email address and app password.