Step 1

Ensure you have the following setting in the Admin > System > Settings. This ensures the customer can still checkout for out of stock products.

Step 2

Now, go to Admin > Catalog > Products. Edit the products for which you want to restrict checkout and enable “Do not allow checkout when qty is 0“. Click Save.

Step 3

Now, try testing to place an order for those products. You should not be able to reach checkout page, it should always redirect to the cart page.