So this plagued me for a while when working on an assimp model loader with animations. Pretty much my animations were screwy and didn't look correct, it literally took me until I found this random blog post where I was able to figure it out: https://devtalk.blender.org/t/export-fbx-the-armatures-rotation-and-scale/6054
When I export and I check off FBX units scale in the scale dropdown in the fbx export settings, things work as indended but static objects are rotated 90 degrees in pitch, so I have to check "apply transform" as well while exporting.