Step 1
Ensure you have the following setting in the Admin > System > Settings. This ensures the customer can still checkout for out of stock products.
Step 2
Now, go to Admin > Catalog > Products. Edit the products for which you want to restrict checkout and enable “Do not allow checkout when qty is 0“. Click Save.
Step 3
Now, try testing to place an order for those products. You should not be able to reach checkout page, it should always redirect to the cart page.