This answer is directed at early stage startups, when you're in the business of finding out product/market fit. Shaving 5ms may increase revenues by 1%, but unless that 1% is a factor more than your yearly compensation, read on.
First, maximize distance between you and managing any infrastructure. Don't open a data center, don't buy servers, don't lease them, and don't even run EC2 instances or mess with EBS volumes. Out-source your infrastructure.
Second, when switching costs are low, make quick decisions. If it takes an hour to deploy your first app on a hosted platform, do the math: it takes an hour to switch to a different provider. Don't worry about picking the best, or what will happen when you grow. Decide early, decide often.
With that in mind, here are the services we use for development and in production right now. If you're wondering where to start, or not happy with one of your providers, check this list:
http://github.com/ for hosting code, documentation, and issue tracker. Developer docs are in Markdown and checked into source control; avoid the Wiki. If you need a friendly UI for editing docs, have a look at http://prose.io.
http://huboard.com/ is a Kanban-style board using Github issues as tasks. Good for small teams.
http://codeship.io is our CI. It picks up changed from Github, runs the test suite, deploys to production, and notifies us via email/HipChat.
http://hipchat.com/ is our water cooler, 1:1 chat, and we also feed it with notifications from Github, CI build status, production alerts, etc.
http://papertrailapp.com/ for logging.
http://rollbar.io for catching and reporting errors. You want errors emailed to you, but don't need 1,000 copies of the same error, and should be able to tell if error persists after deploying new code. There are many exception tracking services, Rollbar does a great job of balancing capabilities with simplicity.
http://pingdom.com/ will monitor if your server is up. Tip: setup alerts for multiple pages, check content not just status code, and monitor your HTML, CSS **and** JS assets.
http://deadmanssnitch.com/ will track your back-end jobs. Basically, jobs ping an HTTP endpoint on completion, DMS alerts if it hasn't seen a ping in a while. You can set different check rates for hourly jobs, daily jobs, etc.
http://pagerduty.com/ so your most important alerts get immediate attention, and for managing on-call rotation. You can integrate it with anything that can send an email (Pingdom, DMS, Rollbar, etc).
http://mixpanel.com/ for application/business metrics. If you need deep Web analytics (e.g. slice by screen resolution and OS version), or SEO, then Google Analytics is where you go. But if you're looking at feature usage, funnels, retention, customer segmentation, etc then MixPanel is the easier path to insight.
http://nodejitsu.com/ for deploying you Node.js Web app in production. I spent 5 minutes on this decision, because switching costs.
http://iron.io for back-end processing. Typically Web app platforms don't want to run your back-end jobs. iron.io offers queuing, including easy integration with Webhooks, scheduling, and place to run your workers.
http://mongohq.com/ for database. Note, you can provision a database directly from the Nodejistu Web console, but that only gives you app/shell access. If you create an account directly with MongoHQ, you get access to the wonderful Web console (great for sharing queries with the PM). I suspect this it the case with other platform/database add-on combos.
http://name.com/ as the registrar because it's OK, and as cheap as they come without being GoDaddy. Amazon Route53 for DNS because the UI is better, there's an API, and it integrates well with S3 and CloudFront, which we use for serving assets.
http://heroku.com/ is the granddaddy of Platform as a Service, but gets pricey fast and never quite grew on me. Regardless, it's often the easiest way to start, and if nothing else, when looking for a hosted service provider – maybe you need a Redis instance or a Graphite dashboard — start by checking out the Heroku add-on page.