Three years ago, I had an interesting problem. It was necessary to assemble a platform to combine multiple racks of servers into a single entity for the dynamic allocation of resources between sites, written for the LAMP platform.
However, the budgets were very less so expensive solutions such as Cisco Content Switch or disk shelves with fiber optics were not affordable.
And, besides, of course, in case if one server is down – this should not affect the operation of the platform was my main concern.
In my school time, I read somewhere that “Necessity is the mother of invention”, which is fairly true.
First of all you need to share a platform into subtasks. You have to do something for the synchronization of data as a shared drive is available. In addition, it is necessary to balance the traffic and have at it some statistics. Finally, the automation of providing the necessary resources – is also quite a serious problem.
Let’s start from the beginning…
I had a choice on what to organize a platform. OpenVZ or XEN ? Each has its pluses and minuses. OpenVZ has a lower overhead, work with files and does not block devices, but cannot run anything other than Linux’ovh distributions. XEN allows you to run on Windows, but more difficult work. I’ve been using OpenVZ, as this is more suited for the task, but you can choose the one you like, there is no restriction on choice.
Then I shared the server space for the VDS, one for each core. Servers were different, and therefore I had a set from 2 to 16 and virtual ok on each server. In the “average house” turned out about 150 virtual ok on the counter.
How to synchronize the data?
The next item – this is the early establishment of VDS on demand + protection against breakage of any server. The solution was simple and beautiful.
Each VDS creates the initial image as a file on the LVM partition. This image “spreads” on all servers in the platform. As a result, we have a backup of all the projects on each server (paranoid cry of emotion), and the creation of a new VDS «on demand” has been simplified to a snapshot image and it start the VDS literally in few seconds.
Database and API
If the integrity of the files were all simple, here’s a sync base things were worse. From the beginning I tried a classic example – master-slave, and collided with a classic problem: slave lag behind master.
The next step was to Mysql-Proxy. As a sysadmin, this was very easy to set and forget, but the configuration should be updated while adding / removing new VDS. But developers have had their own opinion. In particular, the fact is that, it is easier to write a PHP class for synchronization of INSERT / UPDATE / DELETE queries than to learn Lua, without which the Mysql-Proxy is useless.
Their work produced a so-called API, which was able to find neighbors of a broadcast sync up to date and to inform the neighbors of any changes to the database.
But still worth exploring Lua and make native mode, where all requests are synchronized with their neighbors.
FreeBSD
Balancer – it can be said that it is a key aspect of the platform. If they fall to balance server, all work will have no meaning.
That is why I used the CARP to create fault-tolerant balancer, choosing FreeBSD as the OS and Nginx as a balancer.
Yes, NLB has been replaced by two weak machines with FreeBSD (marketers in a rage).
And most importantly – how it works
When starting up the platform for each site runs on a single copy and monitor to balanesere watched to ensure that the primary copy has always worked.
In addition, the balancer was installed to analyze statistics Awstats, which provided all the logs in a convenient format, and most importantly – there was a script, polling each VDS via SNMP for its load.
As we remember, I devoted each VDS on one core, so Load Average in a 1 – this is a normal load for the VDS. If LA became 2 or above – the script that creates a copy of the VDS on a random server and put this in its upstream nginx’a. And when the load on extra VDS fell less than 1 -, respectively, all removed.
Summarize
If you take the rack with servers and switches supporting the CARP protocol, to create a ESDS cloud hosting Server, will need to:
- Explore Lua and adjust transparent synchronization across Mysql-Proxy
- Screw the billing account for additional copies of the VDS and traffic
- Write a web interface for managing VDS
- The filling racks with enough amount of four zeros. Compared with the decisions of the brands, where the price of one stand is the sum of six zeros, counts worth.
