I started out asking "What makes a good church website?" and the answers (there are more than 1) aren't much different than the answers to "What makes a good website?" I tried to stay with what could be easily determined and quantified. There are a lot more dimensions that could be explored, but if these are covered, the others are just window dressing.
One of the benefits of using a CMS is the ease it offers in getting a site up and running fairly quickly without a lot of fuss - just fill in the blanks and you're set. The problem, as stated before, is that you do need to visit and consider each of the options you have, not just accept the defaults. I recommend at least an annual audit for these basic items and also some deeper reflection on the goals you have for your web presence. For some additional questions to consider that is geared towards Christian web stewards is the design questionnaire at InternetEvangelismDay.com That survey definitely has a leaning towards outreach sites, but there are still questions that can provide lots of food for thought. Websites also need care and feeding - get to know it, get to know the environment, get to know the technology and get to know the strategies. But, above all, have a very clear understanding of how a website might fit into your church or ministry.
We could go a lot deeper into our own short survey results, but you'd soon get tired of my ramblings - it's best to save the rest for a future article. If you have something to say on this topic, you can leave a comment or post in the forum and join in the dialog.
Blessings!