Hey,
I have quite in depth experience on this matters, I've configured SAML for most services, I've recently setup Github with Azure AD, I've worked with ADFS as well in the past and successfully setup the connection, I would think the problem is either with the metadata configuration, EntityID, Nameid and SSOServiceURL, this would be the most important aspects of the exchange, it may also be about the SSOURL index from ADFS, either way I think I have enough insight into this matter to help out get this running in no time.
Kind Regards!