Hello,
1) You need to obtain server certificate, that can be achieved by openssl
openssl s_client -connect [YOUR-SERVER]:465 -showcerts < /dev/null | openssl x509 -outform DER > server_cert.der
2) From certificate you need to get correct hash, that depend on TLSA type, Mailcow in DNS modal use 3 1 1, so this should work
openssl x509 -in server_cert.der -inform DER -outform PEM | openssl x509 -pubkey -noout | openssl pkey -pubin -outform DER | openssl dgst -sha256
3) Format DNS record
Above output line like
SHA2-256(stdin)= [HASH]
Use hash and add correct type
3 1 1 [HASH]
Also if you are more familiar whit PHP, you can copy/modify generate_tlsa_digest() function from mailcow it self mailcow/mailcow-dockerizedblob/70126e1f0cb0f78d99446453521b8d665fe07db9/data/web/inc/functions.inc.php#L632