On the VXTIndia blog there's a recent post that gives you a very complete guide to setting up a VPS server running Ubunut with all of the software you'd need to get a (more than) complete server up and running.

Every time you purchase a new Linux VPS, you need to go ahead and set it up for use. Even though we manage to do one server a month, we always seem to forget one thing or the other. So we decided to write down the things that we do. I thought it would be a good thing to share it with everybody as well, so that we could get a few comments about what we’re doing wrong, and people who do it the first time can probably pick up a few things from here.

Steps and software involved in their process include:

• Updating to the latest Ubuntu packages
• Creating other users for the system
• Installing the web environment (including PHP, MySQL, phpmyadmin and Apache2)
• Minin for monitoring
• Configuring IPTables
• Installing Fail2Ban

All of this comes complete with the additions you'll need to make to configuration files and the commands to get everything installed (via aptitude).