It is not trivial.
1. Select your app
2. Click on Rights and Pricing
3. There is a link 'Unless you select specific stores, your app will be for sale in all App Stores worldwide.', click on the 'specific stores'
4. Deselect all locations
Now, after removing the app form sale, you can remove it.