"I know it was really hard for him to feel helpless and not be out there with his team," Kelly tells PEOPLE of the Duke star ...