Archive for category hardware
Server Donation Entry Period Ending
Just to let folks know, we have had quite a large interest in our donation of some of our decommissioned servers. In fact, I have way too many emails!
So to be fair, rather than just stop today, we will stop accepting submissions for this next Monday, September 28th. That means if you want your proposal/request in the running, you have to have it emailed to servers@wikimedia.org by Midnight GMT this coming Sunday, Sept. 27th.
For ease of reference, here is a copy of the post from the start of this process:
It is that time again. We have approx 35 servers to donate to a good home. These are servers that Wikimedia has used on the projects for 3+ years, so they are out of warranty and just not fast enough for us to keep using on the cluster.
The servers will go out to homes for folks who are willing to pay for the freight. They are as follows:
- Dual CPU 2.5 GHz AMD
- 3-4GB RAM Each
- Most have 80 GB or larger HDD
Disclaimers: The Wikimedia Foundation does not guarantee the operation or use of these servers in any shape or form. They are old, some may have dying fans, bad hdd sectors, and the like. Servers have been wiped of information, and they ran through that, but no promises on function!
If you would like to receive some of these servers for your NONPROFIT use, please email servers@wikimedia.org. Please include in your email how you will be using the servers, and the address they would be shipped to. We will review all requests and try to fairly pick out where they go. (Selection process may be refined, but it also may just include throwing darts at a board to break up ties.)
Additions: Due to request, the servers are indeed located in Tampa, FL USA. Zip code 33602 for shipping purposes. This means that if you are international, shipping this hardware is really not cost effective for you. If you want to be in the running still, and are comfortable with personally handing all customs, duties, export, and tax issues, go ahead and email us.
Correction: Dates were off.
PMTPA Router Reboot – Scheduled Downtime (Resolved)
Our primary router for the pmtpa cluster had to be rebooted today at 12:00 GMT. A line card had died and needed replacing, and the
system required a reboot for it to fully take effect. Once that finished, CentralNotice was adding a lot of overhead and had to be disabled for our caching cluster to catch up. Then the overload caused the primary database master for S3 to overload, and we are in the process of switching database masters to another server.
If all went as planned, this would have been a quick 5 minute router reboot and back online. Unfortunately, things do not always work smoothly, so what would have been 5 minutes has been awhile. This post will be updated as more details are resolved.
Update: We have switched database masters successfully and all sites and projects should once again be fully functional as of 14:13 GMT.
PDF export service temporarily down (fixed)
Wikimedia’s PDF export service is temporarily down; the server failed to reboot after a routine kernel upgrade. It should be resolved or replaced with a spare box within a couple hours…
Update: Server is back online.
Intermittent media server load problems
We’ve been seeing some general slowdowns in our image and media file serving recently, including some instances in the last couple days where the sites as a whole have been affected to the point of extreme slowness or temporary inaccessibility.
Domas believes this is related to this reported problem with NFS performance when ZFS snapshots are active. We’ve had some luck so far with it improving after dropping older snapshots (possibly along with restarting NFS and temporarily disabling the image scaler servers to give it a little breathing room to reset).
We’ve been planning for some time to redo the way we access our media files internally which can help reduce the impact on the rest of the site when load problems on the file servers occur, but we might also be able to spread out the load among multiple servers to improve things even more.
Updates will come as we get things back on track…
Update 2009-07-15: We’re temporarily shutting off uploads while we apply the ZFS fix patch and reboot the main file server. You may see some missing images or funky error messages for a little bit, but the sites should otherwise continue working normally until the file server is back up.
Update 2: Server is patched and uploads are back online. This should resolve our performance problems while we continue rearranging the upload servers to be more future-proof.
WMF needs additional datacenter space
The Wikimedia Foundation is looking into the option of expanding into a new datacenter. Currently the plans are tentative, but are expected to become much firmer once discussions with various Datacenter Providers takes place.
Currently, the servers for the projects reside in Tampa, Florida, USA, and in Amsterdam, Netherlands. We actually have moved the servers recently in Amsterdam. Now the time has come to move/expand in the US. We are looking at moving to an area OUTSIDE of Florida, where every single hurricane season is not the cause of distress.
We are currently looking in the Virginia and DC areas, but are not adverse to other areas given the space/power/transit issues. I have already been in contact with a number of vendors, but that doesn’t mean I do not want more options.
Things we require:
- We are looking for Datacenters that offer co-location services with 24/7 access.
- We also require racks have both primary and redundant power drops, from different feeds and circuits.
- The drops also need to be 3phase 208V power.
- Offers a low cost out of band access for our mangement network ONLY (no production traffic.)
- Some kind of NOC in residence in the event of ‘horrible end of the world’ happenings and we need remote hands. (We have LOM and remote reboot capabilities, but having a NOC is never horrible.)
Any interested sales folks at a datacenter can email me rob at wikimedia dot org. Put ‘Datacenter Relocation Project’ in the subject so I am sure I see it!
Also, any folks out there who have decent recommendations, let me have em!
English Wikipedia brief outage
We had a crash on our database master for English Wikipedia. Domas is restarting it and swapping it out for another master server; should be back online in a few minutes.
In the meantime, Wikipedia in other languages and all other Wikimedia sites remain unaffected.
Update 23:39 UTC: We’re back! Looks like approximately 25 minutes of breakage.
An out-of-memory condition on the database master server ended up killing the MySQL daemon…
Pretty Servers
Since we now have our own blog, we can do neat stuff like point out that we have photos of one of our new datacenter deployments. They are tagged on Commons with the Category of Wikimedia Servers. However, if you just want to see the new photos you can do that here. Keep in mind all these are possible due to the generous donations of our readers!