What is a good way to overcome this issue?

Let ye who have no sin cast the first stone. Remember that your life is between you and God. You cannot control what anyone else says or does. What you can do is accept the peace that peth all understanding. Strive to be more christlike and forgiving. When they see you becoming more godly, perhaps they will be attracted and moved to do so themselves.