A major new version of vBulletin (the software that is used for the JSF forums) has just been released. There are some really cool new features in this new version that I think you all are going to enjoy a great deal. Just to name a few of the new features:

User picture galleries with user comment facility
User-created social groups with invite only and moderated membership options
Extended member profile pages
Customizable member profile pages with admin-controlled styling abilities
Inline editing of custom user profile fields
Lightbox viewer for attached images
Reciprocal friendships between users
Public visitor messaging on profile pages with 'conversation' feature
Viewable and comparable history maintained for post edits
User change history
Thread tagging
Search cloud / tag cloud
Social bookmarking integration
Much, much more!Don't worry if you don't know what some of that stuff means; there will be resources posted to learn about the new features, and the forum will continue to work pretty much as it always has for those of you who curl up into a tight ball and break a cold sweat at the thought of change.

While I could upgrade the forum software as it sits right now, there are some recommended support software packages that also need to be upgraded for optimal performance. All these upgrades are non-trivial, and I'm not comfortable performing them on the live servers: the chances of something breaking are pretty good.

So, all of this leads to taking the next logical step, and that is replacing the aging hardware that runs JSF and the JSF forums, and also the underlying Operating Systems. I've been giving it some thought, and I've decided to scrap everything and give JSF a complete hardware, O/S and software overhaul. This is going to be a huge project, it's going to be expensive and it's going to take some time.

The first step is I need to purchase two brand new servers. I am still working out the hardware details, but these new servers will be many, many times more powerful than the current servers, they will each have several times the amount (and speed) of RAM installed in the current servers, the new servers will all utilize hot-swappable drives in a RAID 1 configuration for true redundancy and enhanced reliability and there will also be redundant power supplies and redundant network cards.

For the O/S I'll be using Linux, of course: either Debian, Ubuntu LTS Server Edition or Gentoo. I'm leaning towards Ubuntu or Debian.

Most of this work will be happening in the background, and will not cause any downtime. First I'll need to get the servers ordered. Once the servers arrive I'll work on them in my spare time here at the house. After I install the O/S & and all the support software, I'll need to get everything configured and tested. After that I'll shut down the live forums, import the existing data into the new system, run some tests and then launch the new JSF! The day of the launch should be the only day with any real downtime.

This new system will provide a much faster and more enjoyable experience for everyone, and it will greatly increases our ability to handle large amounts of traffic.

I want to thank everyone who has supported this site financially; these major upgrades would simply not be possible without you!

If you would like to contribute, now would be a great time to do so. Details on upgrading your membership (http://forums.johnstonefitness.com/payments.php) can be found here (http://forums.johnstonefitness.com/payments.php).

I welcome suggestions and comments from my fellow techies with regards to hardware and software.

Here is the hardware configuration for the two new servers:

CPU 1: Quad Core Intel® Xeon® E5420, 2x6MB Cache, 2.5GHz, 1333MHz FSB
CPU 2: Quad Core Intel® Xeon® E5420, 2x6MB Cache, 2.5GHz, 1333MHz FSB
16GB 667MHz (8x2GB), Dual Ranked DIMMs [UPDATED]
PERC6i SAS RAID Controller, 2x4 Connectors, Int, PCIe, 256MB Cache, x6 Backplane
2x146GB 15K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug Hard Drives [UPDATED]
2U Rackmount case
Redundant Power Supply with Y-Cord
2xIntel® PRO 1000PT Dual Port Server Adapter, Gigabit NIC, Cu, PCIe x4The web server (Apache/PHP) and the backend server (MySQL) will have identical hardware.

If you're anything like me you'd like Debian better than Ubuntu. Ubuntu is great for a desktop distribution but it just rubs me the wrong way for servers. They pack it full a bunch of crap that shouldn't be there in a fresh install. Debian is a great vanilla platform to start out on.

Gentoo on the other hand... I guess it's nice if you don't mind compiling everything (not that you're not already aware of that). It's caused me more headaches when a package breaks or fails to compile than anything else. Now days it's just not worth the hassle. You wont get any extra performance benefits compiling everything like you would have a few years ago.

What kind of hardware are we talking here? Have you considered virtualization? Buy one nice server with 16GB of RAM and go nuts. VMware Server and ESX (especially) make me :drool:.

EDIT: I see you just posted hardware details. With another 8GB of RAM that would make one hell of a VM host. :nod:

Make sure u have enough bits and bioses for everything. :nod:

If you're anything like me you'd like Debian better than Ubuntu. Ubuntu is great for a desktop distribution but it just rubs me the wrong way for servers. They pack it full a bunch of crap that shouldn't be there in a fresh install. Debian is a great vanilla platform to start out on.
This would be my first use of Ubuntu Server. I'd heard the LAMP install of Ubunto is pretty clean. That's not the case?

It's been years since I've messed with Debian. How the PMS these days?


Gentoo on the other hand... I guess it's nice if you don't mind compiling everything (not that you're not already aware of that). It's caused me more headaches when a package breaks or fails to compile than anything else. Now days it's just not worth the hassle. You wont get any extra performance benefits compiling everything like you would have a few years ago.Yep. The current servers run on Gentoo, and while I once thought it was a solid distro, it's sort of lost its way over the years. I'm ready to give it the boot.

What kind of hardware are we talking here? Have you considered virtualization? Buy one nice server with 16GB of RAM and go nuts. VMware Server and ESX (especially) make me :drool:.

EDIT: I see you just posted hardware details. With another 8GB of RAM that would make one hell of a VM host. :nod:
I like VM, too, and I think it's a good way to go in some cases but I really like running two physical servers. If one fails, I can use the single server to keep things running.

I'm about to edit the hardware specs. I'm going to bump to 16 Gigs of RAM and lose the SATA drives for 15k SCSI drives. :D

Make sure u have enough bits and bioses for everything. :nod:
Don't get all technical on me please! :o

BTW, thanks for yet another donation (M@ already had a fully paid Platinum membership, and he just added another year to it). :)

This would be my first use of Ubuntu Server. I'd heard the LAMP install of Ubunto is pretty clean. That's not the case?

It's been years since I've messed with Debian. How the PMS these days?
Ubuntu Server is a nice distro but I wouldn't consider it production quality. Its standard LAMP install is pretty clean compared to others but at the end of the day it's just a pre-configured Debian where they try to guess what you want. They try to make it too easy... which is good for beginners. Probably not the greatest for advanced users. I'm not comfortable with some of their default configuration settings either.

Debian's PMS is great. Apt simply rules. The older package manager (dselect or whatever the hell that piece of crap was) is still there if you want it but Apt is the core. I swiched back to Debian (from Ubuntu Server) when Etch was released. In fact Ubuntu uses the exact same thing (again, it's just a glorified Debian). I would definitely play around with both of them before making a final decision.

Ubuntu Server is a nice distro but I wouldn't consider it production quality. Its standard LAMP install is pretty clean compared to others but at the end of the day it's just a pre-configured Debian where they try to guess what you want. They try to make it too easy... which is good for beginners. Probably not the greatest for advanced users. I'm not comfortable with some of their default configuration settings either.

Debian's PMS is great. Apt simply rules. The older package manager (dselect or whatever the hell that piece of crap was) is still there if you want it but Apt is the core. I swiched back to Debian (from Ubuntu Server) when Etch was released. In fact Ubuntu uses the exact same thing (again, it's just a glorified Debian). I would definitely play around with both of them before making a final decision.
Awesome, that's just the kind of feedback I was hoping for. Thanks, Josh. I'll definitely give Debian a strong look.

I'm about to purchase the two new servers, configured exactly as I indicated in my second post. :spaz:

Before I pull the trigger, does anyone else have input relative to the proposed hardware specs?

Before I pull the trigger, does anyone else have input relative to the proposed hardware specs?

Get a case with flashing neon lights and a cowbell :spaz:

The new servers have been ordered! :spaz:

have you thought about putting a publishing system on your homepage? it'd be nice to be able to post comments on your daily news updates or be able to subscribe to feed.

have you thought about putting a publishing system on your homepage? it'd be nice to be able to post comments on your daily news updates

You can probably get (or build) a VBulletin plugin that displays a "news" page based on the contents of a particular sub-forum. I know it's available for some of the PHP-based applications.
That way, you post a new news item to the sub-forum that you specify in the plugin (in this case, "Daily updates" or something), and the front page would then update accordingly, simultaneously allowing people to comment on it directly in the forums.

Just a thought :)

(agree on the RSS feed) :D

have you thought about putting a publishing system on your homepage? it'd be nice to be able to post comments on your daily news updates or be able to subscribe to feed.
I have, but a comment system on the main page would create some overlap with the forum, which is presently used (to pretty good effect) to discuss my daily updates.

It's a subject that deserves more discussion, but that is something apart from all these upgrades and can be added later. The project I just started is going to keep me busy for quite some time!

You can probably get (or build) a VBulletin plugin that displays a "news" page based on the contents of a particular sub-forum. I know it's available for some of the PHP-based applications.
That way, you post a new news item to the sub-forum that you specify in the plugin (in this case, "Daily updates" or something), and the front page would then update accordingly, simultaneously allowing people to comment on it directly in the forums.

Just a thought :)
An idea worth looking into for sure, but it's going to have to be addressed down the road a bit.

I like VM, too, and I think it's a good way to go in some cases but I really like running two physical servers. If one fails, I can use the single server to keep things running.

I'm about to edit the hardware specs. I'm going to bump to 16 Gigs of RAM and lose the SATA drives for 15k SCSI drives. :D


I think VM for your web/application servers is pretty nice - your backup consists basically of moving a single file to another server, making a few IP changes on the host and you are essentially done.

VM for the database side is usually asking for trouble (IME) - the amount of disk I/O and memory that a highly used database requires usually takes it to the point where virtualization doesn't make sense.

Of course, I'm pretty biased since I'm a DBA by trade and hate to see any apps at all on a DB server.

Depending on usage, you might want to consider keeping all the DB files on a separate RAID array (on another controller, too) altogether so you aren't competing with the OS. You may also be better off with Raid 5 than 1, depending on the read vs. write %.

Of course, I'm pretty biased since I'm a DBA by trade and hate to see any apps at all on a DB server. Yeah, the database server is dedicated to that one task. Other than a few administrative/security packages (iptables, SSH, Apache for PMA) that's all the server does.

In the event of a complete server failure, the other server can step in and perform all duties as a short-term solution until the bad server is repaired or replaced.

Depending on usage, you might want to consider keeping all the DB files on a separate RAID array (on another controller, too) altogether so you aren't competing with the OS.That's a bit overkill in this case.

When did we stop speaking English around here?

I like to fancy myself "computer literate"...do I have to rethink my position since 90% of what's been said in this thread is foreign to me?

Great work John, you do great things around here and to know you are still motivated to upgrade and grow is a wonderful thing! :)

That's a bit overkill in this case.

You should see the server cluster I spec'd out for the new 911 center. :D

I'd have to echo the sentiments about using Debian for the OS on the new servers. I've been using Debian on production servers for a little over five years now and I love it.

Having said that, I'm not the one going to be administrating it. I just wanted to fly the Debian flag.:D

I would recommend Ubuntu. Perhaps they do make it easy, but isn't that how it should be?¿ Everything is fully reconfigurable, as is any Linux distro, if anything turns out to cause a problem for you. Remember that there are two different types of Ubuntu: one built for desktops and the other built for severs. (Actually there are many different types but let's not get pedantic).

We're arguing over two types of Debian here anyway. You should be able to use debian packages on an Ubuntu server, but not necessarily vice versa. Ubuntu is by far the most popular Linux distribution at the moment and the one where you'll need to do the least compiling from source if you're installing any strange applications.

To declare an interest, I deleted Windows on my laptop and desktop and now use only Ubuntu desktop edition.

Unless you really need in at the guts of the system, I'd recommend Ubuntu.





. . . and it's got wobbly windows that rulez!! lololol!!!111 (jk :) )

John, I just sent along a few dollars to help with the upgrades. If there's any left over, please order the machine that goes ping. Thanks.

arCITMfxvEc

I'd have to echo the sentiments about using Debian for the OS on the new servers. I've been using Debian on production servers for a little over five years now and I love it.

Having said that, I'm not the one going to be administrating it. I just wanted to fly the Debian flag.:D

I would recommend Ubuntu. Perhaps they do make it easy, but isn't that how it should be?¿ Everything is fully reconfigurable, as is any Linux distro, if anything turns out to cause a problem for you. Remember that there are two different types of Ubuntu: one built for desktops and the other built for severs. (Actually there are many different types but let's not get pedantic).

We're arguing over two types of Debian here anyway. You should be able to use debian packages on an Ubuntu server, but not necessarily vice versa. Ubuntu is by far the most popular Linux distribution at the moment and the one where you'll need to do the least compiling from source if you're installing any strange applications.

To declare an interest, I deleted Windows on my laptop and desktop and now use only Ubuntu desktop edition.

Unless you really need in at the guts of the system, I'd recommend Ubuntu.





. . . and it's got wobbly windows that rulez!! lololol!!!111 (jk :) )
Thanks for the additional input, guys!

Ive been administering Unix/Linux servers since the early 90s, and have tried just about everything under the Sun (no pun) over the years: Irix, OpenBSD, NetBSD, Free BSD, SCO Unix, Debian, Redhat, Gentoo, Slackware, Ubuntu, SUSE...

Each of those operating systems/distros have their strengths and weaknesses. I'm certainly not unaccustomed to going under the hood. In fact, that's where I spend most of my time everyday. I don't like it when things are needlessly complex, but at the same time I despise it when something is dumbed down. I like 100% control, I like lean, I like powerful...

It sounds like they both deserve a look. I've not tried the latest Ubuntu LTS server, and it's been many years since I've played with Debian. I promise that I will give them both a try. :nod:


John, I just sent along a few dollars to help with the upgrades. If there's any left over, please order the machine that goes ping. Thanks.Much obliged, Phil. :tucool:

I'm really excited about this hardware and software upgrade. We're going to have some awesome new features to play with, and it's going to be FAST! :spaz:

I'm seeking input from fellow Apache administrators, especially those who have experience running large forums--preferably vBulletin forums.

I've always used Apache's tried-and-true prefork Multi-Processing Module, but I am considering compiling Apache on the new web server to use the worker MPM. I think a threaded MPM will scale far better and use less resources than a process-based server. Everything I'll be using along with Apache is threadsafe, so I don't foresee any issues.

Pros? Cons? Stability concerns? Should I consider alternatives to Apache, such as lighttpd? Thoughts?

EDIT: Hrm, the PHP team specifically recommends against using PHP with Apache 2.x when it is compiled with a threaded MPM, and their argument (http://us2.php.net/manual/en/faq.installation.php#faq.installation.apache2)make s sense to me. Even though I'm pretty sure everything I'll need to use is threadsafe, it's probably better to be safe than sorry. I guess I'll stick with prefork.

Processes have a heavier footprint than threads, but at least the new web server will be beefy enough to handle a very large number of simultaneous requests. :nod:

You clearly know what you're doing John and as for sys admin, you've got bags more time served that I have. But for the few years that I've been doing this properly there are a few practices that I apply as standard when setting up sites and the servers they're on. Although I would say that most of the sites that I've worked on don't come near to the volume of traffic that you're getting :D.

I've had a lot of success serving up static content through lighttpd, which is proxied through Apache. This means you can fine tune each server for slightly different roles of serving static content and the parsing of php.

Then there is the subject of caching ( memcached for example) and saving compiled code in memory ( Zend Platform, APC to name a few), they can all add some extra zing to serving up application data. Although I'm not too sure that cacheing in the sense of memcached will be of any use in an environment as dynamic as a forum.

I was at presentation with Nate Koechley of Yahoo last year and he talked about the twelve rules for high performance websites ( audio (http://www.htmldog.com/atmedia2007/highperformancewebpages.mp3), slides (http://nate.koechley.com/talks/2007/atmedia-london/high-performance-web-sites.pdf)) . Some of it may not be applicable here as the application (In this case vbullitan) is already developed, but some might.

I've also found the yslow extention for firebug ( a firefox extention) to be of particular use when calculating the performance of a server from a users perspective. The grade it spits out may not mean much in the grand scheme of things, but it is good at picking up on potential tweaks like gziping static content and sending out the correct caching headers for things like css, js and images.

I know the new servers will be built to take a battering, but squeezing every last drop of performance out of them can only be a good thing.

Having said all that I would be hastened to point out that by trade I'm a developer, not a sys admin. I only got into sys admin on webservers because no one else was doing it, and I'm the kind of person that will step up to a challenge when I need to.

Again, I do hope that I'm not telling a granny how to suck eggs, but there might be some useful stuff in there. Anything that I give back to this community is a bonus for me, I owe it a lot.

Must dash, the dogs are looking at me with that tone of voice that translates to "put the laptop down and take us out for a walk dammit"..

I've had a lot of success serving up static content through lighttpd, which is proxied through Apache. This means you can fine tune each server for slightly different roles of serving static content and the parsing of php. I should probably research lighttpd a bit more. I know it excels at serving static content, but I'm unclear on its performance/memory/CPU usage numbers as compared to Apache (prefork) when it comes to serving dynamic content. 99% of what's served at JSF is dynamic, so I've little use for improving static performance.

If I do stick with Apache, I've pretty much decided against the worker MPM. I've been reading some horror stories. It seems PHP is not as threadsafe as the developers would have us believe. I don't want to take chances just to save a little RAM. It's not like I skimped on the RAM (each server will have eight times more RAM than the exisitng servers, and the new RAM is much faster, too), so there's plenty of memory to go around.

Then there is the subject of caching ( memcached for example) and saving compiled code in memory ( Zend Platform, APC to name a few), they can all add some extra zing to serving up application data. Although I'm not too sure that cacheing in the sense of memcached will be of any use in an environment as dynamic as a forum. Caching makes a HUGE difference with vBulletin. I started off using Turk-mmcache, then switched to eAccelerator and, finally, I switched to APC, which I've found to be very fast and quite stable.

On the new server I'm going to give Xcache a try, which I've heard is even faster and more stable than APC.

I was at presentation with Nate Koechley of Yahoo last year and he talked about the twelve rules for high performance websites ( audio (http://www.htmldog.com/atmedia2007/highperformancewebpages.mp3), slides (http://nate.koechley.com/talks/2007/atmedia-london/high-performance-web-sites.pdf)) . Some of it may not be applicable here as the application (In this case vbullitan) is already developed, but some might.I'll check that out this afternoon. Thanks for the link.

I know the new servers will be built to take a battering, but squeezing every last drop of performance out of them can only be a good thing.:nod:

Again, I do hope that I'm not telling a granny how to suck eggs, but there might be some useful stuff in there. Anything that I give back to this community is a bonus for me, I owe it a lot.I appreciate your input! Thanks for taking the time to post. :)

Must dash, the dogs are looking at me with that tone of voice that translates to "put the laptop down and take us out for a walk dammit"..
I know that look well. :nod:

Hello, John, I'm just interesting about your web-server current stats - how many hosts/hits per day do you have ?

Thanks, Oleg.

...Caching makes a HUGE difference with vBulletin. I started off using Turk-mmcache, then switched to eAccelerator and, finally, I switched to APC, which I've found to be very fast and quite stable...
Just curious, but do you remember when the last switch took place?

I'm no expert on the subject by any means, just wondering if the change lines up with what I saw as a significant speed increase around the forums a while back. :)

Hello, John, I'm just interesting about your web-server current stats - how many hosts/hits per day do you have ?Those traffic stats vary quite a lot from one day to the next. The reason is because links to the main site and, to a lesser extent, the forums are constantly getting posted in blogs and other message boards--some of which are extremely popular. The servers need to be able to handle the most extreme loads.

What's extreme (for this site)?

I recall just last spring when the main site was posted on several very high traffic sites. We had almost a quarter of a million unique visitors (220,000) and something like 12,000,000 hits over a 48-hour period! The day after that we broke the record again with another 129,000 unique visits in just 24 hours. Those numbers would have been higher, but the servers kept crashing. Also, those figures don't include the forums, which were also slammed.

I'm not sure that the new hardware will even be able to handle that kind of traffic without slowdown/crashes. :scared:

I'd like to find out, though. :evil:


Just curious, but do you remember when the last switch took place?

I'm no expert on the subject by any means, just wondering if the change lines up with what I saw as a significant speed increase around the forums a while back. :)We've been using caching almost since day one. While caching makes a huge difference in overall performance, the increase from one type of caching to another is probably not drastic enough for most individual users to have noticed. You may have noticed when we added a second server, or when I doubled the RAM in both servers up to 2 Gigs (I'm sorry, but I don't recall the dates). The 2 Gigs of RAM in the old single-CPU servers will be dwarfed by the new servers: 16 Gigs of RAM and dual quad-core CPUs in each server. :)

Yeah, the database server is dedicated to that one task. Other than a few administrative/security packages (iptables, SSH, Apache for PMA) that's all the server does.

In the event of a complete server failure, the other server can step in an perform all duties as a short-term solution until the bad server is repaired or replaced.

That's a bit overkill in this case.

John,

Sorry for the unsolicited input but it's just too exciting that my favorite fitness forum is run by a person who can spell vi and du ;).

Have you looked into circular replication for keeping the backup server current for such an event. If not, it's pretty slick and simple with MySQL. We run two of our forum servers each with mysqld and identical master copies of the database. There is no lag and it's waaay unlikely to lose even one post. The servers don't even breath hard and our load is significant. We aren't vBulletin, but use two other synced (and thus heavily hacked) forum packages, php, apache, FreeBSD so the environment is not totally different. The extra impact of both machines being masters and slaves is virtually unfelt on either system and both have their own unique normal "day jobs" aside from being the fallback for each other.

If you aren't doing that and want to, I have examples within my setup. I did it based on some onlamp articles. It's actually all of a half hour of setup once it's understood.

We just keep the low-change items like php programs and such rsync'd nightly. The two solutions together are a very simple way to handle fallback without reload. I'm suspicious of complex solutions.

Chris

Sorry for the unsolicited input but it's just too exciting that my favorite fitness forum is run by a person who can spell vi and du ;).
:nono:

vim ftw! :D

Have you looked into circular replication for keeping the backup server current for such an event. If not, it's pretty slick and simple with MySQL.Not quite, but almost. The two servers have all the packages installed that would be required if one had to step in and fulfill the other's duties. The only thing I'd need to do is change a few config files, upload and uncompress the databases tarball onto to the web server (assuming the database server failed), or upload and untar the web files onto the database server if the web server fails.

If you aren't doing that and want to, I have examples within my setup. I did it based on some onlamp articles. It's actually all of a half hour of setup once it's understood.Anything to make my life easier. Let's see what you've got!

We just keep the low-change items like php programs and such rsync'd nightly. The two solutions together are a very simple way to handle fallback without reload. I'm suspicious of complex solutions.I'm with you there: simple but powerful is my moto. :nod:

Thanks for the input! :)

The first server ("Iron") will be the web server (PHP/Apache) and the second server ("Steel") will be a dedicated MySQL server. Here are the salient details:

- CPU 1: Quad Core Intel® Xeon® E5420, 2x6MB Cache, 2.5GHz, 1333MHz FSB
- CPU 2: Quad Core Intel® Xeon® E5420, 2x6MB Cache, 2.5GHz, 1333MHz FSB FSB
- 16GB 667MHz (8x2GB), Dual Ranked DIMMs
- PERC6i SAS RAID Controller, 2x4 Connectors, Int, PCIe, 256MB Cache, x6 Backplane
- 2x146GB 15K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug Hard Drives
- 2U Rackmount case
- Redundant Power Supply with Y-Cord
- 2xIntel® PRO 1000PT Dual Port Server Adapter, Gigabit NIC, Cu, PCIe x4



:drool:

Love the server names!

Finally, servers with real hardware. 8 cores :spaz:. None of this sissy single/dual core stuff like at other sites.


User-created social groups with invite only and moderated membership options


I'm curious about this one. Does invite-only necessarily mean they can/will be private? If so, does the forum administrator (you) have access to these areas by default? Otherwise it seems like an open invitation to rule breaking.

I recall just last spring when the main site was posted on several very high traffic sites. We had almost a quarter of a million unique visitors (220,000) and something like 12,000,000 hits over a 48-hour period! The day after that we broke the record again with another 129,000 unique visits in just 24 hours. Those numbers would have been higher, but the servers kept crashing. Also, those figures don't include the forums, which were also slammed.



Thanks for stats, John. Another question : why do you prefer to use RAID 1 on your servers instead RAID 5 ?
By costs reasons ? The SAS hot swap hard drives costs too much, but IMHO, the RAID 5 is more flexible, especially when you need to unexpectedly add disk space to your server.

:drool:

Love the server names!:tucool:

Thanks for supporting the site and helping to make upgrades like this possible.


I'm curious about this one. Does invite-only necessarily mean they can/will be private? If so, does the forum administrator (you) have access to these areas by default? Otherwise it seems like an open invitation to rule breaking.
I have little to no firsthand experience with most of the new features, but this is also a concern of mine.

Based on my current understanding of this new feature, what I'm leaning towards doing is only allowing members with upgraded memberships of a minimum level (maybe Bronze+) the ability to create social groups (but any member can join a social group, so long as the group owner approves it). The owner of the social group would be responsible for enforcing the forum rules and would become a moderator of sorts. The group members can still report posts to the main administrative staff if the owner of the social group is allowing posts that violate forum rules.

The reason I'm leaning towards restricting who has the ability to create a social group is because paying members have demonstrated that they truly care about this community, and I can be reasonably confident that those members would be good stewards of their social group(s).



Another question : why do you prefer to use RAID 1 on your servers instead RAID 5 ?
By costs reasons ? The SAS hot swap hard drives costs too much, but IMHO, the RAID 5 is more flexible, especially when you need to unexpectedly add disk space to your server.No, cost played absolutely no part in my choice to use RAID 1.

My two main concerns when selecting the RAID configurations for the new servers were SPEED and RELIABILITY.

RAID 1 performance with both reads and writes--especially small writes, which happens a lot here--is superior to RAID 5.

RAID 1 can tolerate the complete loss of one drive with no data loss, and only requires 2 drives. RAID 5 can only tolerate the loss of one drive before data loss occurs, and it requires at least three drives to run, thus increasing complexity (and needlessly so, at least for this particular purpose).

As for hard drive space, with over 4 years experience running this place I've got a pretty good grasp of how much space is required now, and how much will be needed in the future. For example, most would be surprised to learn that every bit of user-generated data on this forum (all the posts, PMs, user account data, attached images, avatars, profile pictures and other kinds of attachments--everything from the past four years) totals less than 3 Gigabytes. Factoring in space for the O/S, the main web site and the log files (which are permanently removed and archived off-site every few months), and we're still less than 30 Gigs.

So, if we've only created 3 Gigs of data in 4 years, and assuming a massive growth rate of an additional 3 Gigs per year for the next 10 years, we're still only going to have 33 Gigs of data + 30 Gigs of other stuff. 10 years is used as an example, but in practice we'll certainly have replaced these new servers by then.

Taking it a step further, even if I did want to replace the existing drives, doing so is very simple under Linux.

Just a note to say the planned updates look fantastic!

Can't offer any tech advice... but I sent along my PayPal dollars. Thanks again for the site and all you do.

:nono:

vim ftw! :D

Not quite, but almost. The two servers have all the packages installed that would be required if one had to step in and fulfill the other's duties. The only thing I'd need to do is change a few config files, upload and uncompress the databases tarball onto to the web server (assuming the database server failed), or upload and untar the web files onto the database server if the web server fails.

Anything to make my life easier. Let's see what you've got!

I'm with you there: simple but powerful is my moto. :nod:

Thanks for the input! :)

aww, those danged new-fangled modern editors... just kidding, I broke down and started using macvim on my workstations, it's seriously productive over my vi stodginess. But I actually do still use vi on the servers, just can't break the habit.

I'm PMing you with a synopsis of how I did the setup of the replication and the link to the more indepth discussion on onlamp.com. It presumes you have a spare NIC on each system and can isolate db (and other non-public) traffic. I really believe in that concept from a security standpoint.

Chris

:drool:

Love the server names!

I think we had some better ones picked out in teh WL. :D

Just a note to say the planned updates look fantastic!

Can't offer any tech advice... but I sent along my PayPal dollars. Thanks again for the site and all you do.
Thank you for your generous support of the site! :)


I'm PMing you with a synopsis of how I did the setup of the replication and the link to the more indepth discussion on onlamp.com. It presumes you have a spare NIC on each system and can isolate db (and other non-public) traffic. I really believe in that concept from a security standpoint.
Got it, thanks. Looks very simple and straightforward.

Yes, the DB server uses RFC 1918 IPs, and sits behind the dual-homed web server on a direct gigabit link. Even thought the database server is not forward-facing, I still secure it as if it were. As some of you already know from reading my home journal, I'm big on security in the real and in the virtual world. :nod:

Thanks again!

That's some serious equipment you have ordered there! I would have upgraded my membership sooner and higher, but I'm moving next month and I'm finally going to get to build my own Crossfit gym. I'm starting with a great 1000+ lb. bar and bumper plates, not cheap stuff.

Thanks for everything you do and while I'm not aiming for bodybuilding at all, this is an awesome site for information, inspiration, and best of all, community!

Thanks, John.

p.s. I like Debian for servers, granted I only do really small scale stuff with LAMP, I just think Ubuntu is riding the hype wagon a bit too hard.

That's some serious equipment you have ordered there! I would have upgraded my membership sooner and higher, but I'm moving next month and I'm finally going to get to build my own Crossfit gym. I'm starting with a great 1000+ lb. bar and bumper plates, not cheap stuff.
Awesome! I wish I had room for a nice bumper plate station in my gym. :dreamy:

Thanks for the support! :tucool:

p.s. I like Debian for servers, granted I only do really small scale stuff with LAMP, I just think Ubuntu is riding the hype wagon a bit too hard.I don't blame them, but I tend to agree with you. Shoot, every other article on digg these days is about Ubuntu (well, that and GTA4). :lol:

I know jack about behind the scenes workings of the internet (I thought it was magical house elves), but I've thrown a few more pennies in the pot, to help out. Better profile pages sounds cool.

I know jack about behind the scenes workings of the internet (I thought it was magical house elves), but I've thrown a few more pennies in the pot, to help out. Better profile pages sounds cool.
:eek:

Dude. You are already a Lifetime Platinum supporter. That's 500 bucks. You've more than paid your "dues". I can't believe you sent more money. As much as I appreciate your unbelievable support of JSF, please... no more. There are over 20,000 members here; you've done your share, and then some.

From all of us, thank you. :)

:eek:

Dude. You are already a Lifetime Platinum supporter. That's 500 bucks. You've more than paid your "dues". I can't believe you sent more money. As much as I appreciate your unbelievable support of JSF, please... no more. There are over 20,000 members here; you've done your share, and then some.

From all of us, thank you. :)

Ah, dont worry about it, I'm happy to help. That reminds me, I can't remember if I ever got the coffee mug, do you have any of those left?

Ah, dont worry about it, I'm happy to help. That reminds me, I can't remember if I ever got the coffee mug, do you have any of those left?
Check your PMs. :)

Dell has already shipped out the two machines that will eventually become the new JSF servers! The servers were not scheduled to ship until mid-month, so this news comes as a very pleasant surprise.

Of course I ordered the servers with no Operating System/empty hard drives, so I've got quite a bit of work ahead of me before they'll be ready. The only data I'll be restoring is the actual web site content and all the forum data; everything else will be installed and configured from scratch.

The new servers should be in my hands by this Friday at the latest. I've got a pretty busy weekend ahead of me, but I'll try to carve out a chunk of time to at least get the project started. At this point it would be premature to attempt to provide an estimated date of completion, but I'll keep everyone posted on my progress in this thread.

OK, as luck would have it this week I had to prepare a new server for work. As such, I've had a good excuse to play around with a few different Linux distributions. Other than a couple of minor niggles, I REALLY like Debian the best. I felt right at home with it, and I love the PMS. It blows Gentoo out of the water. Ubuntu is a solid distro, but ultimately I felt more comfortable with Debian.

Debian will be the O/S I install on the new JSF servers.

The hardware will be here on Friday. :spaz:

Debian will be the O/S I install on the new JSF servers.

Curious, what criteria did you use to evaluate the different distros?

Curious, what criteria did you use to evaluate the different distros?It was very informal. I mainly just installed each distro, played around, ran some tests to see which would break my existing scripts/software the least, checked out the PMSs, evaluated online support--that sort of thing. Ultimately I felt most at home with Debian, and so I went with my gut instinct.

Good choice on the Debian. It's tried and true and the basis for ubuntu anyways.

The hardware looks sick and should have NO problem handling the sites load I would think.

Good choice on the Debian. It's tried and true and the basis for ubuntu anyways.I think I made the right choice. The more I play, the more I like.


The hardware looks sick and should have NO problem handling the sites load I would think.
Yeah, the new servers are overkill for our normal traffic loads, but I wanted to plan for future growth, and also be able to handle the very large spikes in traffic we sometimes see. Even with the new hardware I'm not sure we can handle something like what happened last spring, but that was a very unusual "perfect storm".

Hey, John, good luck with the upgrade. For what it's worth, though, I've never noticed any performance issues with the site. Were there slow response times for users as they posted or something like that?

Anyway, have fun with the install. Although I'm much more of a lightweight when it comes to IT stuff (I'm just a database developer/programmer), I can appreciate how exciting it would be to set up a whole system like that, especially on brand-new high-powered hardware!

Speaking of the servers...they're not being delivered by DHL, are they? :)

What happened last spring? I wasn't around much then.

Hey, John, good luck with the upgrade. For what it's worth, though, I've never noticed any performance issues with the site. Were there slow response times for users as they posted or something like that?During peak hours things have been slowing a bit, and there have been a few instances recently where resources have been strained, causing the database server to become overloaded.

The servers' load averages have been creeping up lately, and we're a little past the threshold I feel comfortable with.

Slow servers and web sites are annoying, and I don't want JSF to be one of those sites. :nod:


Speaking of the servers...they're not being delivered by DHL, are they? :)Thank goodness, no. FexEx, who I'm generally pleased with.


What happened last spring? I wasn't around much then.JSF went sort of viral for a few days last spring, and was posted on several very large sites such as fark, digg, reddit, del.icio.us, and many other sites small and large--all in the space of 2 or 3 days. This resulted in massive amounts of traffic, crashing servers, headaches and frustration for everyone involved.

I talked a bit about this event in more detail (with specific numbers) a little earlier in this thread.

Somehow the the two servers were separated in shipment, but the good news is one of them is here safe and sound! I can only work on one at a time anyway, so no big deal.

Pictures...

Fresh out of the box:
26073


Under the hood:
26074


16 Gigs RAM, 2 Quad Core CPUs, 4 cooling fans:
26075


Powered up in my home office. Ready to start installing:
26076

I talked a bit about this event in more detail (with specific numbers) a little earlier in this thread.My fault, my bad. Admittedly, I skipped some of the longer passages in this thread due to my complete lack of knowledge of computer lingo. Most of that technical stuff was WAY over my head.

Sorry to ask an already answered question. :spank:

I just noticed a couple minutes of downtime. Was that you switching onto the new server?

:D

I just noticed a couple minutes of downtime. Was that you switching onto the new server?
I don't think the downtime was on this end.

I've got quite a bit of work to do before the new servers will be ready. :nod:

I made an incredible amount of progress today!

- Debian is installed, and most of the supporting software/libraries are installed and configured.

- The RAID stuff is working perfectly. I spent a little too much time gleefully pulling the hard drives while the server was hot, and then checking to make sure everything was working properly. It always was. :dance:

- I've got Apache locked down and all the port 80 virtual hosts up and running. I still need to play with SSL.

- The current server runs MySQL 4.0.x, and I've upgraded to MySQL 5 on the new server. I successfully pulled a mysqldump of the existing forum data and imported it into the new server with zero problems. The migration from MySQL 4 to MySQL 5 was my biggest concern, so the successful outcome of this test was a huge relief!

- FTPd, SSHd, PMA, AWS and Postfix are all installed and 85% configured.

- I ran a stress test on the new server. This particular test makes the current web server spike up to a 5.0-7.0+ load average. On the new sever I could not get the load average above .05 no matter what I threw at it. :)

I still have to get iptables configured and work on the SSL stuff, and a couple of my own php programs are slightly broken (the DVD database on twowiresthin.com needs some massaging), but other than that we're looking really good after only a half-day of work. :nod:

If things continue to go this smoothly I may be able to deploy the new servers some time next week!

I'm seeing double right now. It's been a long day, but a good day. I'm going to bed!

I ran a stress test on the new server. This particular test makes the current web server spike up to a 5.0-7.0+ load average. On the new sever I could not get the load average above .05 no matter what I threw at it. :)


That's great, just out of curiousity what do you use to stress test? Do you use apaches 'ab' or something else. This is one area of sys admin, that has always eluded me and I would love to know more about it.



If things continue to go this smoothly I may be able to deploy the new servers some time next week!


Sweet, it's always nice when a plan comes together :)

Just as a leftfield idea...looking back over the initial list of new features I didn't see the vbulletin blog (http://www.vbulletin.com/features_blog.php) mentioned. Do you think that we'll see this in the new version as well?

That's great, just out of curiousity what do you use to stress test? Do you use apaches 'ab' or something else.It was more of a real-world test. I once used a neat little php script to automatically create thumbnails from the images posted in the pictures section of the main site. The script worked pretty well, but it was absolute murder on the servers--especially if more than a few dozen people happened to be loading the picture page at the same time. Last spring when the site got slammed, I learned pretty quickly that the script needed to go. I got rid of the script (the thumbs are no longer dynamically generated), but I installed it on the new server just to compare. Even with several dozen simultaneous loads, the server just shrugged it off like it was nothing. :)


Just as a leftfield idea...looking back over the initial list of new features I didn't see the vbulletin blog (http://www.vbulletin.com/features_blog.php) mentioned. Do you think that we'll see this in the new version as well?Yes, it's very likely that I'll implement that feature. :tucool:

just out of curiousity what do you use to stress test?

That was going to be my question :)

Since at least a couple of you are interested in seeing some standardized stress tests, I'm happy to take a few minutes and provide those for you. :)

Keep in mind that I have not yet done any performance tweaking. I expect to be able to improve on these results.

I'll start with 'ab', since just about everyone who has an apache server has that utility available to them. I'll also keep "top" running in another terminal and report system load averages while the tests are running.

Let's start very light: 1000 requests in batches of 10 requests at a time (I'm using the JSF front page as the test page for all these tests):

iron:~# ab -n1000 -c10 http://192.168.0.200/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.0.200 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests

Server Software: Apache
Server Hostname: 192.168.0.200
Server Port: 80
Document Path: /
Document Length: 17534 bytes
Concurrency Level: 10
Time taken for tests: 0.240681 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 17722000 bytes
HTML transferred: 17534000 bytes
Requests per second: 4154.88 [#/sec] (mean)
Time per request: 2.407 [ms] (mean)
Time per request: 0.241 [ms] (mean, across all concurrent requests)
Transfer rate: 71904.31 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 1 1 1.9 1 12
Waiting: 0 1 1.7 1 11
Total: 1 1 1.9 1 12
Percentage of the requests served within a certain time (ms)
50% 1
66% 2
75% 2
80% 3
90% 4
95% 6
98% 7
99% 8
100% 12 (longest request)


That wasn't much of a challenge. The load average didn't even move from 0.00, and all 1000 pages were served up in less than 3/10th of second each.


Let's increase the challenge exponentially: 10,000 page requests, 100 requests at a time:

iron:~# ab -n10000 -c100 http://192.168.0.200/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.0.200 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests

Server Software: Apache
Server Hostname: 192.168.0.200
Server Port: 80
Document Path: /
Document Length: 17534 bytes
Concurrency Level: 100
Time taken for tests: 1.953154 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 177220000 bytes
HTML transferred: 175340000 bytes
Requests per second: 5119.92 [#/sec] (mean)
Time per request: 19.532 [ms] (mean)
Time per request: 0.195 [ms] (mean, across all concurrent requests)
Transfer rate: 88608.48 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 4
Processing: 4 18 5.7 17 64
Waiting: 1 18 5.6 17 63
Total: 5 18 5.7 17 64
Percentage of the requests served within a certain time (ms)
50% 17
66% 19
75% 20
80% 21
90% 25
95% 31
98% 37
99% 42
100% 64 (longest request)

Still not much of a challenge. 10,000 page requests were served up in less than 2 seconds, the majority of which took 17ms each. The longest request took just 64ms to be fulfilled. The server barely noticed: the load average peaked at 0.01.


I'm going to increase the load exponentially once again: 100,000 page requests, 1,000 requests at once:

iron:~# ab -n100000 -c1000 http://192.168.0.200/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.0.200 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests

Server Software: Apache
Server Hostname: 192.168.0.200
Server Port: 80
Document Path: /
Document Length: 17534 bytes
Concurrency Level: 1000
Time taken for tests: 26.715080 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 1772200000 bytes
HTML transferred: 1753400000 bytes
Requests per second: 3743.20 [#/sec] (mean)
Time per request: 267.151 [ms] (mean)
Time per request: 0.267 [ms] (mean, across all concurrent requests)
Transfer rate: 64782.29 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 54 659.5 10 21005
Processing: 1 98 610.3 52 24253
Waiting: 0 82 609.7 33 24220
Total: 2 152 936.2 65 24267
Percentage of the requests served within a certain time (ms)
50% 65
66% 69
75% 71
80% 73
90% 76
95% 81
98% 793
99% 3063
100% 24267 (longest request)

As you can see, this test got the server's attention! It served up 100,000 pages--1,000 requests at a time--in less than 30 seconds, but it sure had to work for it. The load average shot up to 5.98, but there were no failed requests. Even at that load, 95% of the pages were served up in less than 1/10th of a second. Also, less than 1 Gig of the available 16 Gigs of RAM were needed. Not too bad.

Out of curiousity, what's the load average out of? what's the units?

Out of curiousity, what's the load average out of? what's the units?
It's magic.

Seriously, that's a complicated question. It shouldn't be, but it is. Here's a summary from a much longer multi-part article (http://www.teamquest.com/resources/gunther/display/5/index.htm) that attempts to demystify "load average" under Linux:


So, what have we learned? Those three innocuous looking numbers in the LA triplet have a surprising amount of depth behind them. The triplet is intended to provide you with some kind of information about how much work has been done on the system in the recent past (1 minute), the past (5 minutes) and the distant past (15 minutes). As you will have discovered if you tried the LA Triplets quiz, there are problems:

The "load" is not the utilization but the total queue length.
They are point samples of three different time series.
They are exponentially-damped moving averages.
They are in the wrong order to represent trend information.

These inherited limitations are significant if you try to use them for capacity planning purposes. I'll have more to say about all this in the next online column Load Average Part II: Not Your Average Average.


Personally I use it to see when things are starting to go bad. I have a terminal open to my right at all times that shows the JSF servers' load average. When it's less than 3 or so, all is well. Once it starts getting above that, things start slowing. Around 15+ all hell breaks loose. :scared:

Of course the numbers I just mentioned are based on the existing servers, which are single-CPU systems. As I understand it, load averages should be divided by the number of processors on a multi-cpu system. The new servers have 2 quad-core CPUs. I'm unclear on if that means I divide by 8 or divide by 2 in order to get a reasonable approximation compared to the old servers.

Just for fun and comparison, I thought I'd duplicate the same tests I just performed on the new web server on the current web server. The results are not pretty.

1,000 connections, 10 at a time:

iron:~# ab -c10 -n1000 http://www.johnstonefitness.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.johnstonefitness.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests

Server Software: Apache/2.0.58
Server Hostname: www.johnstonefitness.com
Server Port: 80
Document Path: /
Document Length: 15450 bytes
Concurrency Level: 10
Time taken for tests: 24.884536 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 15664064 bytes
HTML transferred: 15475688 bytes
Requests per second: 40.19 [#/sec] (mean)
Time per request: 248.845 [ms] (mean)
Time per request: 24.885 [ms] (mean, across all concurrent requests)
Transfer rate: 614.68 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 36 45 95.5 41 3048
Processing: 167 201 16.6 199 369
Waiting: 40 49 9.4 47 128
Total: 213 247 97.9 240 3260
Percentage of the requests served within a certain time (ms)
50% 240
66% 244
75% 247
80% 248
90% 255
95% 271
98% 332
99% 374
100% 3260 (longest request)

Ugh! A mere 1,000 page requests at the very low rate of 10 requests per second took almost as long as the new server took to serve 100,000 pages at the rate of 1,000 simultaneous requests. The new server didn't move from a 0.00 load average, but the existing web server peaked at a 2.96 load average. That load average does includes normal traffic (which is very light ATM), but still....

That's pretty pitiful performance for such a light request.


Here's 10,000 page requests, 100 at a time:

iron:~# ab -c100 -n10000 http://www.johnstonefitness.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.johnstonefitness.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests

Server Software: Apache/2.0.58
Server Hostname: www.johnstonefitness.com
Server Port: 80
Document Path: /
Document Length: 15450 bytes
Concurrency Level: 100
Time taken for tests: 215.113645 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 156566792 bytes
HTML transferred: 154683596 bytes
Requests per second: 46.49 [#/sec] (mean)
Time per request: 2151.136 [ms] (mean)
Time per request: 21.511 [ms] (mean, across all concurrent requests)
Transfer rate: 710.77 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 39 511 739.5 377 9425
Processing: 375 1629 1023.4 1489 16971
Waiting: 161 551 842.5 398 16186
Total: 417 2140 1270.9 1875 19049
Percentage of the requests served within a certain time (ms)
50% 1875
66% 1935
75% 1985
80% 2037
90% 3185
95% 4599
98% 5914
99% 7857
100% 19049 (longest request)

The new server performed the above test in less than two seconds. The existing server? It took a whopping 3 minutes 35 seconds! That's like four quadrillion percent longer. Also, on the existing server 95% of the requests took up tp 5 full seconds to complete. The new server? 95% of the request were completed in less than 3/100th of a second.


I probably shouldn't even run this final test on the existing server, but here it goes: 100,000 page requests at the rate of 1,000 requests at once...

iron:~# ab -c1000 -n100000 http://www.johnstonefitness.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.johnstonefitness.com (be patient)
apr_socket_recv: Connection timed out (110)
Total of 8086 requests completed

Ha, the server stopped responding before it could even complete 10% of the test.

Yeah, things are going to be much, much faster around here real soon now. It's not an exaggeration to say the new web server is literally 100 times faster than the old web server.

Yeah, things are going to be much, much faster around here real soon now. It's not an exaggeration to say the new web server is literally 100 times faster than the old web server.

So, is it safe to say that our connections will be the limiting factor for speed?

hey john, have you bothered to look into services like Amazon's Web Services (AWS), their EC2 (cloud computing) and S3 (storage service)? It's something I've been playing with recently for some organizations I work for and they seem great in-so-far-as to avoid expensive hardware costs. As there seems to be a growing set of methods to distribute the load of heavily-used web servers/services, I'm interested in others thought on utilizing such tools.

hey john, have you bothered to look into services like Amazon's Web Services (AWS), their EC2 (cloud computing) and S3 (storage service)? It's something I've been playing with recently for some organizations I work for and they seem great in-so-far-as to avoid expensive hardware costs. As there seems to be a growing set of methods to distribute the load of heavily-used web servers/services, I'm interested in others thought on utilizing such tools.
I looked at S3 awhile back, and I also briefly looked at EC2. I was turned off by the costs of EC2. One extra large instance running 24/7 (one extra large instance is what most closely mirrors just one of our new severs) put the costs around $500.00/mo--and that's the price for computing power that only approaches the power of one of our new servers. Two instances would double those monthly costs.

Once you add in data transfer costs (which, of course, is a significant monthly expense) the numbers just didn't work out.

Progress update & discussion of the new blog feature here (http://www.johnstonefitness.com/news/may_2008_news_archive.php#05122008). :)

John,

When the forum changes are ready to "go live" can you post an idiots guide / laymans term explaination of what the changes are and what it means. I am pretty computer savvy but i have been lost by many of the recent updates on the main page, and sometimes felt like i was reading a textbook! :p If i was lost with my IT level of knowlege, there is bound to be some people here completely flumoxed and perhaps even put off by the changes, simply because they dont understand them. Many people just know the basics of the internet, how to log on, post a reply to a forum etc. They would probably be 100% confused by the recent weeks posts.

The updates seem to assume everyone has your background and knowlege of IT. RSS feeds, pingbacks etc probably mean nothing to many people. :nod:

Is a pingback when you try and fire a rubber band at someone and it backfires? ;)

John,

When the forum changes are ready to "go live" can you post an idiots guide / laymans term explaination of what the changes are and what it means. I am pretty computer savvy but i have been lost by many of the recent updates on the main page, and sometimes felt like i was reading a textbook! :p If i was lost with my IT level of knowlege, there is bound to be some people here completely flumoxed and perhaps even put off by the changes, simply because they dont understand them. Many people just know the basics of the internet, how to log on, post a reply to a forum etc. They would probably be 100% confused by the recent weeks posts.

The updates seem to assume everyone has your background and knowlege of IT. RSS feeds, pingbacks etc probably mean nothing to many people. :nod:


I sincerely apologize! :o

I've tried to keep the updates on the main page as nontechnical as possible. I attempted to save the more technical banter for the discussion thread. Unfortunately it seems that my sense of perspective is a little skewed in this area. Again, I'm very sorry if I've added to anyone's confusion.

The forum will continue to work just as it always has, so there will not be a learning curve for anyone who's not interested in the new features (or simply wants to pick up the new stuff at their own pace).

That said, I will be posting detailed information on all the new features for anyone who is interested in learning about them. New terms will be explained, and all the new software features will be completely demystified. I assure you, none of it is complicated, and it's unfortunate that I may have created the opposite impression by inadvertently peppering my updates with a little too much technical jargon. I'll try and do better. :)


Is a pingback when you try and fire a rubber band at someone and it backfires? ;)
Pingbacks/trackbacks are two different forms of linkbacks. Linkbacks simply allow authors of blogs (or other web documents) to know when someone else links to their work. :)

More on linkbacks (http://en.wikipedia.org/wiki/Linkback).

Excellent stuff John, the blog feature is something that I'm really looking forward to.

No problemo, no need to apologise. :)

Excellent stuff John, the blog feature is something that I'm really looking forward to.
I've been playing around with it on the new server and it's really, really cool! :tucool:

If you're anything like me you'd like Debian better than Ubuntu. Ubuntu is great for a desktop distribution but it just rubs me the wrong way for servers. They pack it full a bunch of crap that shouldn't be there in a fresh install. Debian is a great vanilla platform to start out on.

Gentoo on the other hand... I guess it's nice if you don't mind compiling everything (not that you're not already aware of that). It's caused me more headaches when a package breaks or fails to compile than anything else. Now days it's just not worth the hassle. You wont get any extra performance benefits compiling everything like you would have a few years ago.

What kind of hardware are we talking here? Have you considered virtualization? Buy one nice server with 16GB of RAM and go nuts. VMware Server and ESX (especially) make me :drool:.

EDIT: I see you just posted hardware details. With another 8GB of RAM that would make one hell of a VM host. :nod:


I couldn't agree more.. I use Ubuntu as a client OS, but never as a server OS.

Are concerned that it will become harder to enforce the rules with all of these new features like blogs?

Are concerned that it will become harder to enforce the rules with all of these new features like blogs?
Not at all. Problematic users are rare here, because they never last long.

Quite a few of the members do an extremely good job of bringing any issues to my attention, and the new features won't change that.

Hi John, sounds like you are having fun with all the new kit. I just wanted to post and give my appreciation for all the hard work you do with JSF. I know that this site has definitely changed my life for the better :)

I also have a question. Will the new blog system be able to integrate smoothly with the current membership level system? What I'm getting at is, if I have a blog instead of a journal thread, will I still be able to have it in a "private" members-only sub-section?

Servers look like the 2950's. They are rock solid if so. I manage a VMWare farm racked with them and they are as good as the HP DL380 and the like. Good choice.

I also have a question. Will the new blog system be able to integrate smoothly with the current membership level system? What I'm getting at is, if I have a blog instead of a journal thread, will I still be able to have it in a "private" members-only sub-section?That's a great question.

The answer, I believe (I'll let you know if I find out otherwise) is "no, but...". Because blogs are not kept in a particular section of the forum, they are open to everyone by default--however you do have come control over who can view it. Here's a screenshot of the "privacy" section of the blog control panel:

_________________________________________

26117

_________________________________________

Further, you can choose to moderate all comments, or even stop comments from being posted altogether.

I think it would be cool to let the blog author choose which usergroups can view his/her blog, and I've asked vB about that functionality. Maybe it will happen in a future release or someone will create a hack. I'll keep you posted with any news.


Servers look like the 2950's. They are rock solid if so. I manage a VMWare farm racked with them and they are as good as the HP DL380 and the like. Good choice.
Yep, Poweredge 2950 IIIs :tucool:

I love OMSA! I wrote a shell script so I get paged if anything at all is amiss. It's neat to open the case or pull a hot drive and have my pager go off seconds later alerting me to that fact. :)

The addition of blogs is nice. I might finally start a journal here on JSF. It is not relevant for me, but those who have a journal already will probably be interested in some sort of data migration plan.

The addition of blogs is nice. I might finally start a journal here on JSF. It is not relevant for me, but those who have a journal already will probably be interested in some sort of data migration plan.
Well, threads are not blogs, so there really won't be any data migration. People can abandon their journals if they want and link to their new blog, they can stay with the traditional thread-based journal, or they can keep a blog and a journal if they want. :)

I mean, it's not like I have any means to convert an existing linear thread into a blog. :nod:

I'm sure you're already aware of this but there were some pretty important software updates today (OpenSSL) that you might want to implement before going live. :)

I'm sure you're already aware of this but there were some pretty important software updates today (OpenSSL) that you might want to implement before going live. :)
God, I know... I wound up spending half the day working on actual real work instead of my pet project. :mad:

:lol:

God, I know... I wound up spending half the day working on actual real work instead of my pet project. :mad:

:lol:
Yeah, that little patch caused me so many headaches today. Had to generate new keys for every single server. My favorite was when OpenVPN failed to start because it detected vulnerable keys. :blank: :rolleyes:

There are about 600 people on the forums right now (plus a few hundered viewing the main site) and the web server's load average is 0.09. :eek:

With the old web server it would have been between at least 4, and probably closer to 10.

From the main site to the forums, everything is noticeably faster. Very noticeable. The images that provide the "framing" for the forums would always trickle into loading when I viewed/refreshed a thread. Not anymore!

From the main site to the forums, everything is noticeably faster. Very noticeable. The images that provide the "framing" for the forums would always trickle into loading when I viewed/refreshed a thread. Not anymore!
Thanks for the feedback!

I agree, browsing the forum is much more enjoyable now.

The server load is still so low that I keep checking to make sure it's actually serving pages. :lol:

I mean, it's not like I have any means to convert an existing linear thread into a blog. :nod:


I could think of hacking up some script that would convert the owner's posts into, well, posts, and other people's posts into comments to the corresponding posts, and let people use this script to convert their journals, but that's up to you. Anyway, very smooth migration, and great blogging feature, congrats!

John, definitely faster. I love it! Nice work man!

I could think of hacking up some script that would convert the owner's posts into, well, posts, and other people's posts into comments to the corresponding posts, and let people use this script to convert their journals, but that's up to you.
I think it would be extremely difficult to create something automated that would handle stuff like multi-quoting cleanly. It could turn what was once a very readable thread into a real mosh.

Anyway, yeah... I think I'm cool with just moving forward with blogs as a new feature, and not trying to turn a cat into a dog. ;)

Anyway, very smooth migration, and great blogging feature, congrats!Thanks!! :)

I love how that in less than two weeks you've taken a huge project and made it happen. When I read the first post in this thread and your update on that day, I assumed this would take at least a month...if not longer.

Great job John!

John, were you aware that the site was having issues this morning around 7am CST?? Several PHP errors on the main page. Just thought I'd let you know in case you missed it. Glad to hear the hardware/software updates overall went well though. :tucool:

I love how that in less than two weeks you've taken a huge project and made it happen. When I read the first post in this thread and your update on that day, I assumed this would take at least a month...if not longer.

Great job John!
Thanks! :)

John, were you aware that the site was having issues this morning around 7am CST?? Several PHP errors on the main page. Just thought I'd let you know in case you missed it. Glad to hear the hardware/software updates overall went well though. :tucool:
Yes, that error was caused by me. :o

I fixed it, it won't be a problem again. :)

I've noticed the speed change too. :)

I work for yahoo and spend my days in their latest and most cutting edge datacenter, but I'm a technician and work mostly with hardware and basic immaging/cloning problems. What is the most efficacious way to learn how to use the this hardware as it relates to network traffic and how the network infrastructure handles the requests?

I'm impressed with how smoothly your site runs John, great job!

What is the most efficacious way to learn how to use the this hardware as it relates to network traffic and how the network infrastructure handles the requests?
Sorry, I'm not sure I understand what you are after. :o

Hardware without software is useless, so I assume you are asking how to configure things on the software side to best take advantage of the hardware? It's a pretty open-ended question. What specific task(s) are you interested in learning about?

I'm impressed with how smoothly your site runs John, great job!Thanks! :)

Hi John, great site!

Just wondering if you've given any thought to using Solaris 10. It will run on your hardware and there are a lot of really nice features like ZFS - which kicks ass - and DTrace. It will run all your software and it is free as well. You could also look at FreeBSD as well, which if a also a very good and stable OS