I have never had this happen and need some quick guidance. A check drawn on another financial institution was deposited at my institution on 8/30/24. On 9/5/24 the check is returned by the paying bank as NSF. Is this a late return that we can refuse to accept?

Let's count days.

Deposit 8/30/24 meaning that the Federal Reserve received the check on 8/31/24. Due to the holiday, the drawee bank received the check on 9/3/2/4 and has until midnight the next business day to return the check as NSF. The Federal reserve receives the retuned check on 9/4/24 and you bank receives it on 9/5/24.

It is unlikely that this is a late return.