You can’t let go until you admit you are the one that is holding it.
In order to let go of pain or sadness or grief or anxiety (or any other “it”), you need to admit it is you that is holding it in place, and forcing it to stay around.
Once you let go, it leaves.