Things to consider

Choosing a company to host your website or choosing to do it yourself is very difficult. There are a lot of things you need to take into consideration. Different companies offer different services and different levels of service and knowing the difference between them can mean your site loading quickly or slowly and can save you thousands of dollars.

The price of a good host

I've seen the exact same basic website hosting package sell for anything from free to 40$ per month. Be careful to check out and compare pricing plans when you're looking for a host. Especially be wary of free plans. They may say "unlimited bandwidth" but that's really limited by how fast their old server can send out responses over a slow internet connection while simultaneously hosting hundreds of other sites. Free plans come in another version like GoDaddy's where the plan is free and only kinda slow but they put ads up on your site and you can't take them down. Your users are not going to like that. For a basic site you'll look at paying around 7-20$ per month with a good host. Managed hosting will cost a little bit more than that. And if you're getting hundreds of thousands of visitors per month you should be looking at about 150–400$ per month for hosting. Keep in mind these figures also vary a lot depending on if you need extra memory or hard disk space or such.

Hosting it yourself

If you want to host the website yourself you need to have someone around who really knows web server maintenance. A web server is a very complicated machine running a number of programs all of which take extensive configuration. You're going to want to run a Linux distribution with Apache, Tomcat or Nginx if you don't want to get hacked. Choosing the right machine to run this software on isn't that tough given that you can run a website off of almost any modern computer. However if you're expecting lots of traffic or are running something very professional you may want to consider a professional-grade server or at least a more powerful machine.

Your server will also need a good internet connection. Today you can frequently get the level of speed and connectivity you need from an average home or business internet connection. If you have a business internet plan, you can and should get a static IP address. You'll need that to make sure your server is always accessible at the same location. If you plan on hosting your server in your home and don't want to pay for a business-level internet connection, you'll need to use a service like DynDNS that lets you hop IP addresses with less downtime.

Virtual Machines and Colos

Another way to have full server control but not worry about the machine or your ISP is to purchase a virtual private machine from a service like Amazon's EC2 or to stick your real server in the same building with other people's machines(usually called a Colo). If you have quickly changing needs or are looking to experiment a little before committing to a certain setup, virtual machines, especially cloud based ones, are the way to go. You can rent them by the hour for pennies, they always have great internet connections and you can upgrade them from 1 core to 4 cores or double your RAM in a couple seconds. This is especially handy if you need separate servers for database, storage, balancing and caching and don't want to pay for 7 different physical machines.

WordPress Specific

In as much as finding a good website host in general is hard, finding one for something specific is harder. If you're looking for managed hosting for WordPress(and you should be unless you have an expert tech guy) I would recommend going with either WP Engine or Page.ly. They are the big two managed WordPress hosting companies and they both provide excellent service. Both of these companies have hosting environments tailored to the needs of WordPress so your site will run a lot faster than normal. Personally, I host my site and all my clients' sites with WP Engine and I would highly recommend them. Out of all the companies I've ever dealt with, they have the best customer service. I'm a really annoying customer and they're always great to me and go out of their way to help me fix anything that's broken or explain things about hosting that I don't understand. You should seriously check them out.

Another way to go about managed hosting is to get REALLY managed hosting where you basically get to choose a color and start posting. Companies like WordPress, tumblr and Shopify all offer you the option to just buy a domain name and have them handle almost everything else. The obvious downside is a lack of control and flexibility but at least you don't have to think about it at all.

Other aspects and features to think about

If you have managed hosting or are renting a virtual machine or something similar, make sure your website host offers access to a CDN(Content Distribution Network). As I see it, all good website hosts do that. If for some reason you don't have access to one, get one. You can get monthly or bandwidth-based plans from big CDNs today for very little money. A CDN will take your files and distribute them to high-power servers closer to your users so your site seems to load a LOT faster. It's an absolute must today.

Even when using a CDN, try to get your main site files hosted as close to your target audience as possible. It can cut seconds off of your site's load time to have it hosted within a hundred miles of your users as opposed to across the globe.

Pay careful attention to what happens to your email. If you're hosting yourself, you might want to host your own email servers. Personally, I have my email set up through Google Apps and I couldn't be happier. It's simple to set up, familiar to use and always works.