I guess it depends on the nature of the rejection and how “persistent” the asker is.
Just because there’s no direct physical or psychological harm doesn’t make it okay though. Harassment doesn’t require those things. Consistent unwanted advances would qualify.