Speeding Up Your WordPress Blog with W3 Total Cache (W3TC)

Introduction

Most of us mortals, you know the kind that aren’t making six figures from our humble little blog, are using a shared server that is cheap. Well, it’s cheap for a reason. Lots of other sites are crammed on the same server and share the resources. When your site isn’t very popular that’s rarely a problem, but as you start to grow your traffic, performance really counts. And, if you hit the jackpot of blogging, a mention on a really influential site like Digg (“the Digg effect”), Lifehacker or Huffington Post, your site will be inundated with traffic that your poor server can’t handle. Luckily, there is a great solution that doesn’t require much money or technical expertise: a caching plugin. Oh, by the way, that statement I just made about it not being important when your site isn’t popular. That’s not really true either. Why? Because Google uses site speed as part of their ranking algorithm. And, various studies have shown that longer site load time has a negative affect on various customer behaviors (time spent on site, purchases, etc.)

For a long time I was using the WP Super Cache plugin for WordPress and then I read an article about CloudFlare, originally referenced on the Travel Bloggers Facebook group. In that discussion, Michael Tieso recommended using W3 Total Cache (W3TC) with Amazon CloudFront. All of this got me investigating the difference between WP Super Cache and W3TC. Basically, making a very simplified argument, Super Cache is fairly simple to setup with relatively few configuration options, and thus is quite useful for the average Joe who isn’t that comfortable with technology or the nuts and bolts of their WordPress blog. W3 Total Cache on the other hand is much more powerful, having been created by the CTO of Mashable.com for use with their incredibly popular site. The downside is that it can be a challenge to configure and many people don’t bother or do it incorrectly, thus seeing little to no real improvement. If you’re interested in reading some thoughts about these two popular cache options, check out the article, W3 Total Cache Plugin for WordPress Eats WP Super Cache’s Lunch!.

Before investing time in setting up and using W3 Total Cache, you might find this great video on using W3TC useful to see an overview of what can be achieved.

Setting Up W3 Total Cache

There are several good guides around the Web for setting up the W3TC plugin, including this referpages guide and this wpbeginner guide (the latter is more comprehensive). Zemalf has also written an excellent step by step tutorial which, though written for an older version, will still be helpful. Some guides go into details about setting it up using a CDN, including another referpages guide, one from digital inspiration, and this one from Tony Hue. I recommend you check them out to become even more comfortable with the plugin and process, but if you just want to get going, here’s the basic rundown.

First, install W3 Total Cache and activate it. You will now see a tab labeled Performance in the Navigation region (left-hand sidebar) of your WordPress admin panel, with the available parameters that can be set. You can also reach the settings page from within the Active Plugins section of your WordPress admin panel. Now let’s look at the important parameters.

General Settings

1. There is a compatibility check option that you can use. Most likely if you have WordPress up and running with no problems on a reputable hosting company you won’t have any worries.
2. By default the preview mode is active. Don’t change this until you feel that your setup is complete.
3. Enable the Page Cache option. Leave the default enhanced disk cache method.
4. Enable the Minify option. If you will be using a CDN you should select the Manual minify mode option. Leave the default disk cache method and the other defaults unless you have a specific reason to change them.
   
5. Enable the Database Cache option. This should help your site load time because WordPress sites heavily use a database and the required queries are often what slows yours site down or crashes it with high traffic.
6. Using the Object Cache option depends on your server. If you are on a shared hosting account the response time of the disk may not be fast enough to use the “disk” method. Test this option with and without database cache to ensure that it provides a performance increase.
7. Enable the Browser Cache option. This will turn on HTTP compression and add expiration headers on the objects of your site. When someone visits your site their browser will store a copy of it so that on the next visit the site does not have to re-load entirely (which obviously speeds things up). Each browser resets the cache differently, even if a file has not been changed. By enabling this option your site will tell the browser how long to hold each object. For example, your favicon doesn’t change very often so that is an object which can be stored for one month period without change.
8. The next option is CDN (Content Delivery Network). This is recommended as it can make a huge improvement in performance, but it is also the one thing you will probably have to pay extra for (there is a self hosted option, however). Since this is a bit more complicated than just toggling an option box, I will discuss it in detail later.
9. The next section deals with Varnish. If you’re not already using it there is no reason to start so just ignore this section.
10. The next section deals with CloudFlare. I will discuss that in detail separately so keep reading.
11. Finally there is a Miscellaneous section. I personally enabled the Google Page Speed dashboard widget (first having to register to obtain an API key) because I want to be able to see my site performance every time I visit my WordPress admin panel.
    

Page Cache

Go to the Page Cache page and more or less select all options in the General section, though there probably is no real need to cache the 404 pages or search strings so maybe leave those unselected, and unless you have e-commerce capabilities you probably aren’t using SSL either so you can leave that option unselected as well. If you need to change any of the advanced settings you probably don’t really need to be reading this in the first place. In other words, leave those sections alone.

Minify

Make sure the option to rewrite URL structure is selected. Use your own judgment to decide if you want to receive error notices and/or disable for logged in users. In the HTML and XML section select all except the Don’t Minify Feeds option (though you could select that if you see a reason).

Use the help wizard (listed in a sentence at the top with a relevant button) to decide which javascript and CSS files should be minified. This wizard will identify most of the files you will care about but by default they won’t be selected for minifying so you will have to do so. Typically, minification of advertiser code, analytics/statistics or any other types of tracking code is not recommended so keep that in mind. In general, it is not at all intuitive (to me at least) that this step is necessary so make sure you do this. After you select all relevant files to minify they will appear listed in the respective JS and CSS options sections. For the CSS section you should probably select all options except Combine Only. Leave the Advanced section alone unless you have a specific reason to alter it.

Database Cache

You shouldn’t need to modify anything here unless you have specific information to the contrary. If you do need to modify this, most likely you will use the Never cache the following pages option. For example, if one of your page or section heavily relies on database queries, then it is wise to add that page or that entire section in this field.

Object Cache

Again, only modify this section if you have a specific reason.

Browser Cache

I think again it is safest to not modify this section, though there could be good reasons to do so, but do your own research if you think changes might be useful or necessary.

User Agent Groups

Leave this as is unless you have a specific reason to change it.

Referrer Groups

Leave this as is unless you have a specific reason to change it.

CDN

See the next section

Content Delivery Network (CDN)

What’s the point of using a CDN? Basically, it is a system of storing copies of your site in different locations around the planet so when someone far away from your server tries to view your site they get served by a data center most convenient for them, thus speeding up performance. Also, if your site gets massive traffic all of a sudden (say, for example, by being featured on a hugely popular site) the CDN will help handle that traffic and keep your site up and running (the caching features of W3TC will help with this as well, of course).

You’re probably thinking now that a CDN sounds complicated and expensive. Well, not so complicated thanks to the fact that W3 Total Cache has built-in support for multiple CDN providers. But yes, you will have to pay something extra. For a basic blog this probably won’t be too much (less than a few dollars a month), but you pay for bandwidth so the more popular your site the more you will pay. If you don’t like the idea of spending more money on your blog there are three options:

1. Choose to not use the CDN option
2. Choose the Self Hosted option. To do so, you will need to create a subdomain on your own server (and in the process create a matching DNS Zone record). For example, you might have something like cdn.domain.com. You will then need to configure the FTP options in the Content Delivery Network section accordingly. Be sure to upload the appropriate files.
3. Use CloudFlare. See the section below dedicated to this interesting service.

Amazon S3 and Amazon CloudFront

I think the most common CDN choices in the blogosphere will be Amazon Simple Storage Service (S3) or Amazon CloudFront. If you’re wondering about the difference between the two, check out the article, Confused about Amazon S3 vs. Amazon Cloudfront? Don’t Be! The quick take: S3 isn’t really a CDN but rather is Amazon’s cloud storage service. This is where you would store a copy of your website (you can also use S3 for personal backup purposes as well). CloudFront is a CDN but it needs some original source for the content it will replicate across the globe. The easiest source to use is S3 though you don’t actually have to store your content on S3 to use CloudFront. However, you DO need to have an Amazon S3 account to use CloudFront. It seems a bit confusing but don’t worry because you only pay for what you use. Plus, S3 is fairly inexpensive (Amazon does charge a small fee to transfer the data from your S3 storage bucket to all the data centers but the bulk of your bill will be determined by usage).

A logical question you might have is why to use one service vs. the other (or both)? S3 is a cloud storage product but it is not a CDN (that’s why Amazon has CloudFront as well). Still, for many sites the benefits of having your content stored in a high-performance cloud supported by Amazon might be worth the few extra dollars of cost (versus relying on your own server’s hardware which is probably inferior and is being shared with other websites). Using only S3 won’t give you the geographical route and bandwidth optimization that CloudFront or a similar CDN offers but it will still provide some performance improvement.

Configuring S3 within W3TC

So, how do you configure S3? First, create an S3 account. It may take several hours for your account credentials to be functional, though for me it was mere minutes. Next, you need to obtain your Access key ID and Secret key from the Access Credentials section of the Security Credentials page of My Account. Make sure the status is “active.” Next, make sure that Amazon Simple Storage Service (Amazon S3) is the selected CDN type on the General Settings tab, then save the changes.

On the Content Delivery Network Settings tab enter your Access key, Secret key and enter a name (avoid special characters and spaces) for your bucket (Amazon’s storage terminology) in the Create a bucket field by clicking the button of the same name. If using an existing bucket simply specify the bucket name in the Bucket field.

Note that there is an option called Replace site’s hostname with. By default, the items (e.g., images, external files) stored on S3 are going to show in your source code as xyz.s3.amazonaws.com. If you would rather change that to xyz.domain.com you can do so using the CNAME feature, which I discuss below so read that section if interested.

Now, click the Test S3 Upload button and make sure that the test is successful. If not, check your settings and try again. Save your settings.

Finally, go to the General Settings tab and click the Enable checkbox and save the settings to enable CDN functionality. Empty the cache for the changes to take effect.

Configuring CloudFront within W3TC

To use CloudFront, perform all of the steps above for S3, except select the Amazon CloudFront CDN type in the Content Delivery Network section of the General Settings tab (select the pull option).

For the General section of the CDN tab, select all but the last option. You can select the last option as well if you have a reason but I doubt many will have a need.

For the Configuration section, you will enter the Access key ID and the Secret key as described earlier. For the Origin field you can enter a S3 bucket name or you can just use your own server as the origin by entering your site’s URL. If using S3, you can either create the bucket using this W3TC screen or you can do so using the Amazon AWS Management Console.

Just leave the SSL support as “Auto” unless you have a reason to change it.

The field called Replace site’s hostname with should automatically be populated if you created the CloudFront distribution via W3TC. If you create a distribution using the Amazon AWS Management Console instead, just make sure to enter the identifier that will be created that way.

As I mentioned above for using S3, you can enter a CNAME if you wish to have your images and other files referred to as xyz.domain.com instead of xyz.cloudfront.net. To do so, you will need to make appropriate changes to your DNS Zone, which I will explain further on in this guide. In contrast to S3, with CloudFront you can actually specify up to 10 different CNAMEs to use. For basic sites this will not be necessary, but for high traffic sites it could prove helpful.

With the W3TC CDN configuration section complete, make sure that your CloudFront Distribution Status is enabled and State is deployed (using AWS Managemente Console). Now use the Test CloudFront distribution button and verify everything has been setup correctly.

You can probably just leave the Advanced section as it is but if you do notice that certain files on your server are not being handled by CloudFront then you will have to sort of give it a nudge. That’s what the Custom file list is for. Just add any file being ignored to that text box (you can use the * wildcard) and it should be taken are of. By the way, a good method of finding files not being handled by CloudFront is to see the results of a page speed test. See below for more details.

Finally, go to the General Settings tab and click the Enable checkbox and save the settings to enable CDN functionality and empty the cache for the changes to take effect.

Notes:

• If some of this doesn’t make sense to you, the basic tutorial Amazon S3 / CloudFront for W3 Total Cache may help.
• For some reason nobody bothers to mention whether you want to select the Origin Push or the Origin Pull CloudFront option. If you are interested in understanding the difference, read Push Vs. Pull: The Tale of Two CDNs. From a W3TC perspective using push means you must create a S3 bucket to store your site contents to feed CloudFront (and you have to upload your content to Amazon S3 manually). On the other hand, using the pull option will let CloudFront grab data directly from your server. So, unless you have a specific reason to use the push option, it’s probably better to choose pull.

MaxCDN and Other CDN Providers

Although I have concentrated on Amazon, W3TC supports several CDN providers. If you are interested in using MaxCDN check out the wpbeginner guide because they walk you through setting that up. For others I am sure a quick Google search will help you find what you are looking for or perhaps there will be instructions on that provider’s website.

Media Library

At the top of the CDN configuration section there is the option of importing attachments into the Media Library. What is this? The media library import tool is for old or “messy” WordPress installations that have attachments (images etc in posts or pages) scattered about the web server or “hot linked” to 3rd party sites instead of properly using the media library. The tool will scan your posts and pages for the cases above and copy them to your media library, update your posts to use the link addresses and produce a .htaccess file containing the list of permanent redirects, so search engines can find the files in their new location. You should backup your database before performing this operation.

CNAME?

When you start trying to figure out W3TC and content delivery networks, you will regularly come across something known as CNAME. This is basically an alias system used by the Domain Name System (DNS), which is sort of the Internet’s phone book. CNAME helps when running multiple services (like an FTP server and a web server) from a single IP address. Each service can then have its own entry in DNS (like nslookup to confirm that the changes have been correctly propagated.

If you are using Hostgator or another hosting company that uses cpanel, Tony Hue describes how to set up a CNAME or you can see Hostgator’s own How to Create a CNAME record in cPanel?. To summarize those articles, in cpanel you go to the Advanced DNS Zone Editor section and add a record. For type you will select CNAME. For name you will enter the name you would like to use (e.g., cdn.domain.com). CNAME is the CDN address (URL) that you will be trying to hide (e.g., xyz.s3.amazonaws.com). The TTL (Time to live) option is basically how long your information stays valid before being updated on the Internet. Tony talks about setting it to one hour and his graphic shows a “1” but I believe that value is actually given in seconds. I use Hostgator and their defaults are 600 (10 minutes) so I just used that.

CloudFlare

CloudFlare is a bit of a different beast. To get a good overview, read CloudFlare: A website security product accidentally makes sites 60% faster, but for the quick story it is a proxy security service that has over time also evolved into a CDN. By routing all traffic to your site through its network it can prevent malicious traffic and attacks from ever reaching you. Likewise it can prevent spam, bots that want to steal your email address, and other bad activities. The best part: the basic service is free.

Although CloudFlare is also a CDN, it is not as complete as you would find in other CDN providers. For example, it doesn’t cache any HTML which means you won’t see as much performance improvement. Still, you will definitely see improvement which means that in one sense you can use it with W3TC instead of a CDN, especially if you are opposed to spending any more money running your site. At the same time, it can be used in addition to a CDN to provide some extra performance benefits as well as the security protection. And, CloudFlare should lower your CDN bill as well because the static objects will be handled by CloudFlare instead of your CDN. Based on CloudFlare statistics, about 65% of hits get handled by CloudFlare. Mathew Prince from CloudFlare answered a question on this topic on Quora and in his answer he makes the very good point that you should experiment with CloudFlare on its own versus CloudFlare + a CDN to see what works best for your needs. Jan Husdal did that and you can see his results if you’re interested.

So, the short story on CloudFlare is that with or without a CDN, you probably want to give this service a try.

To use CloudFlare with W3TC you first must set up your domain with CloudFlare. I won’t go into those details because their site actually does a good job of explaining things and walking you through the steps. They claim the process can be finished in five minutes, though if you aren’t technologically savvy it will probably take a bit longer. Also, since DNS changes are involved there can sometimes be a delay of up to 24-48 hours, though that is rare. Anyway, once finished, setting it up to be used with W3TC is a very simple process. In the W3TC General Settings tab there is a separate section titled Network Performance & Security powered by CloudFlare. Simply complete the relevant items and enable.

So, setting up CloudFlare is fairly easy, but there are some peculiarities and issues that you should be aware of.

• There is a Development Mode option in the same section where you configure CloudFlare in W3TC. You can turn this on to bypass the CloudFlare cache for up to 3 hours which is useful when you need to make changes to your site (especially if the changes involve graphics, CSS, or javascript). When you are finished you can purge the entire CloudFlare cache so all visitors will see your changes. CloudFlare also creates the direct.yourdomain.com alias (you can change that to something else during setup if you prefer), which bypasses the CloudFlare caching being used on there is a solution with detailed instructions.
• If you are going to use CloudFlare, with or without W3 Total Cache, you should install their plugin. Three main benefits of using the plugin: you will see the correct IP Address information for comments posted to your site, there is a database optimization feature, and comments you mark as spam will be reported to CloudFlare and thus increase the security protection provided for all.
• There should be no conflict between the W3 Total Cache minify and the CloudFlare minify but I don’t see that both are necessary. If you prefer not to use CloudFlare minify you can turn it on or off through the CloudFlare settings page (first visit the My websites section, then the Settings pull down menu, selecting CloudFlare settings. Finally find the Auto minify option and turn it off.
• When you setup CloudFlare it will recommend that you pass all traffic to your public website through its service. One important implication of this is that if you want to FTP into your site for whatever reason, you will need to connect to wiki page on whitelisting their IPs with your website host to fix this.

Testing Your W3TC Setup

One thing I learned from my experience is that whether you fully understand the options or are completely baffled, it is really a good idea to test how your implementation is working. How? There are many tools you can use, including the following:

• Mozilla Firefox + Firefox Developer Tools + Yahoo! YSlow
• Mozilla Firefox + Firefox Developer Tools + Google Page Speed
• Mozilla Firefox + Firefox Developer Tools + Hammerhead
• Google Chrome + Google Speed Tracer
• Pingdom
• WebPagetest
• Gomez Instant Test Pro
• Resource Expert Droid
• Web Caching Tests
• Port80 Compression Check
• A simple online web page compression / deflate / gzip test tool

I personally like Pingdom Tools and WebPagetest, with the former being quick and easy (it tests the load time of all elements on your page and gives you an overall load time with some basic statistics) but the latter giving MUCH more detailed results. I found it useful to use Pingdom when tweaking things and when I finally felt good about my W3TC setup I used WebPagetest to find some remaining performance problems to try and tackle.

In general, I recommend you do a test before setting up W3 Total Cache so you know whether or not you have seen improvements or not. Besides that, if you really want to dig into what W3TC features have an effect on your performance, try enabling them one by one, clearing the cache and doing a performance test.

The output from WebPagetest is pretty thorough, although somewhat confusing at times. Still, it can provide a lot of ideas for further improvements that you can make to your site’s performance. One obvious thing to look for is any file that hasn’t been routed through your CDN. If you find any, you will probably need to add them manually. To do so, go to the Advanced section of the CDN tab and add the files in the Custom file list (wildcards can be used to select and entire directory, for example).

Another thing you can do to test your W3TC configuration is to turn on debug. You can find the debug option in the Debug section of the General Settings tab. Keep in mind that it’s best either to logout or use another browser which does not have an active WordPress Admin session to review the debug data, which you will find at the bottom of the page’s source code. Also, don’t forget to turn the debug mode off after you are finished setting things up.

Finally, if you use YSlow for testing, be aware that it doesn’t recognize all CDNs. If it doesn’t recognize yours, you can add the hostnames to YSlow using Firefox’s preferences. Here are the steps to follow:

• Go to about:config in Firefox. You’ll see the current list of preferences.
• Right-click in the window and choose New and String to create a new string preference.
• Enter extensions.yslow.cdnHostnames for the preference name.
• For the string value, enter the hostname of your CDN, for example, mycdn.com. Do not use quotes. If you have multiple CDN hostnames, separate them with commas.

If you specify CDN hostnames in your preferences, they’ll be shown under the details for Rule 2 in the Performance view.

Handling Plugins with W3TC

One thing you will probably want to do is use your CDN with some of your plugins, which are most likely not going to be handled by W3TC by default. As mentioned in the testing section, you will notice this by looking at the various files listed in the Pingdom or WebPagetest results (the former being simpler for this purpose). Instead of showing a URL of something like domain.s3.amazonaws.com (or whatever CDN you use) you will just see your domain name. To fix this, just go to the CDN section of the W3 Total Cache configuration and scroll down to the Advanced section. There you will see Custom File List with a box to the right where you can add your own custom files. For example, say you are using the sexybookmarks plugin (a popular one). Just add the following line:

wp-content/plugins/sexybookmarks/*

Do this for all the plugins you are using and then save the settings. If you are using S3 you will need to select the upload custom files button in the General section on the same CDN page.

Miscellaneous W3TC Considerations

• In various places in the setup page for W3 Total Cache you will see an option to clear the cache (or clear all caches). This is quite useful especially when you are modifying and testing your settings. It can also be useful if you make a change to your blog, for example an update to the theme. Though keep in mind that CloudFlare has its own cache so you need to purge that separately.
• If you use Amazon S3, check out S3 Fox, a Firefox extension that helps you manage your S3 files and folders.
• If you use the WPTouch plugin, you will need to enter the rejected user agents under the Advanced section in your CDN settings tab.
• Captcha and recaptcha will work fine with W3TC, however you will need to prevent any pages with forms from being cached. Add the page’s URL to the Never cache the following pages box on the Page Cache settings tab.
• Some themes use a script to create thumbnails and, unfortunately, these cannot work with the CDN. This is actually the case with my very old Woo Themes theme and apparently it isn’t that uncommon, though more modern programming techniques don’t use this anymore. You’ll know if you see a bunch of thumb type files in your Pingdom report that aren’t being pulled from your CDN source. If your site used timthumb (or possibly another thumb PHP script), and you really want to beat it into cached shape, one possible solution can be found in the article, Thumbs Up.
• If you don’t have a multiple server configuration, you don’t need memcached.
• This article focuses on using W3TC but Marcus Lindy Sortijas has written a similarly useful document, How to Speed Up Your WordPress Site, that lists a few other optimization ideas worth considering.

Back It Up!

Now that you have everything setup, it is best if you create a backup for your W3 Total Cache Configuration. You can easily do so from the General Settings tab. There is a section for Import / Export Settings. Just click Download to save the settings file from your server to your computer.

My Results

When I was using WP Super Cache my load speed was approximately 7+ seconds. I don’t know what it was before that as I never tested and I don’t feel like testing it now.

When I switched to W3 Total Cache, using the minify and caching options, but no CDN it dropped to about 4.1-4.7 seconds.

When I added CDN using Amazon S3 it dropped to about 2.8-2.9 seconds.

With CloudFront I achieved about 2.5 seconds.

With CloudFront and CloudFlare together I didn’t see any noticeable difference compared to just using CloudFront. If anything it actually got a bit worse, about 2.9 seconds. ‘

I haven’t tested CloudFlare without CloudFront yet.

Like this content? Why not share it?