The reply rating system only checks the first incoming message in a conversation. It starts deducting points after 2 days if you haven't replied in that time. Your rating will improve over time when you respond to other messages promptly.