I think the issue was that when I logged in I had 24 unread notifications. Because the drop down and the notifications page are both limited to 20 items, I couldn't view the oldest 4 notifications. Going to the notifications page tho marked them all read.
We can see if the infinite scroll works well within the popup box, otherwise the only way would be going to the notifications default layout. I think that effort is better be invested in the version 4.4 since all this UI will be discontinued anyway.