Why don't you go upstairs under the pretense of having a headache and asking to borrow some asprins and casually ask if everything is okay while you are there?
If your neighbour asks why, explain that your headache has just made you more sensitive to sound and you could help notice the crying child but you understand this is the sort of thing children do.
If you play it right you will get the sort of reassurance you want without looking like a nosy busy body.