Source: http://www.livefromsiliconvalley.com/venture/startup.html

Form a Corporation or LLC

The best way to do this is to have a service firm do the work for you with the state for a small fee. We have used, and reommend Legal Zoom.

Get a business permit

While the County of Santa Clara issues no licences, the mailtain a great list of all the individual cities and their phone numbers for their business/license permit offices. Check out Santa Clara County business license per page.
Santa Clara has reasonable fees. Check them out.

Get Business Cards

Try: VistaPrint

Get Insurance

A general liability policy is a small investment. We recommend:
Get a Bank Account

WAMU seems to have the best basic business checking, and no fees.
Get yourself a web site

We use 1and1.com and recommend them highly. Good service and reasonable cost, and free tools. You will have what you need fast.