Archive for category software

Wikipedia Mobile October Update

We have a lot of cool stuff that got pushed out today. I’m really excited to tell you about it. First of all, Derk-Jan Hartman (github: hartman) has been hard at work bringing us NetFront support. NetFront is the HTML rendering backend to many devices. Here is a list of the devices that Derk’s hard work has given us blessed access to.

  • Most SonyEricsson Phones
  • Nintendo Wii
  • Sony PS3
  • Sony PSP

He’s also working on Opera Mini support and has done a lot of awesome refactors on the code. Fixing many an embarrassing lines of code for me. Its great to see other people chipping into the project and I know for me personally, its a big inspiration to see people giving their time to help this project out!

subcategoryNow, on to a small tweak that we made to the subcategory expansion system. Its a really small change, but should make using the app even easier. If you want to expand a subcategory, clicking on the title of the category gets the job done. No more having to aim your finger at the “Show” button. Obviously, it also hides if you do it when the section is visible. Its these types of changes that are my favourite. Someone might not even notice it, but it should make their usage of the site a litttttle bit easier.

We’ve also done various and sundry internal changes. We got a patch from Jacques Crocker to use Bundler to manage our Gems. Getting the capistrano part right on that was a bit difficult and so we had a little downtime today while I was working out those kinks. But, from now on, it should be much, much easier to get a local copy of Wikimedia Mobile up and running on your system.

We’ve also expanded the number of supported languages recently. All of this is thanks to the tireless work of Niklas Laxström and others from the TranslateWiki project. They were very helpful in building an API for us to import new languages. I wrote a couple rake tasks and Ruby bits to make it all go. So, now all you have to do is type `rake lang:import` and it will download the freshest strings you’ve ever parsed.

Here is an updated list of all of the languages we support: af, ak, ar, az, bg, bn, br, bs, ca, cy, de, dsb, el, en, eo, es, eu, fi, fr, gl, gv, he, hi, hr, hsb, hu, ia, id, is, it, ja, ka, km, kn, ko, ksh, kw, lb, li, lv, mg, mk, mt, nl, no, oc, pl, pms, ps, pt, ro, ru, sah, sh, sk, sl, su, sv, te, th, tr, uk, vec, vi, wo, and xal. I really have no idea what most of those languages are, but I’m also super happy to support them. And, I’d like to welcome the newest member to our class: Akan (ak), comes to us from Ghana! I know, cool right? Technically, its not a language but an Ethnologue. (Must stop myself from reading too much about this on Wikipedia… and finish this post)

Well folks, that’s it for the moment. Now that I’ve completed my move to another continent you should be seeing a lot more updates coming through on the mobile site. I’m really excited about where we are going and I’m also excited to reveal the results of our survey in the next couple weeks. Some really interesting and exciting things in there!

16 Comments

SVG in Wikipedia and Wikimedia Commons

page1-200px-SVG-Open-2009-Wikipedia.pdfSlides for my talk at SVG Open available for download as PDF or Keynote source. (I can make my test corpus available as well — let me know if interested!)

– brion

2 Comments

Babaco is ready for tasting

Preview of the second set of usability features, Babaco, are available through user preferences. The first feature is the article outline in the right hand side of the editing area. The outline updates itself in real-time as you type the headers in the articles and provides links which when clicked will jump you to the start of each section in the article. The second feature is the assisted way to insert links and tables. Instead of inserting wiki syntax, a dialogue box pops up when you hit the link icon in the toolbar. For internal links, link suggest features offers auto-complete options and validate the existence of articles. A click of the table icon in the toolbar will also assist define the number of rows and columns without modifying table rows and columns in the wiki syntax. Thirdly, new search and replace dialogue is added to the advanced toolbar. These features are released in the stealth mode, which means users need to turn them on by going to user preferences. To enable these features, please go to “My Preferences”, select ‘Editing’ tab and enable the features listed under ‘Experimental features’. We, the usability team, is still refining look and feel, but we wanted to invite active users to start using these features and provide us feedback.
Known Issues: Accuracy of cursor positions using the article outline feature (aka Navigable Table of Contents) degrades in long articles and significant so if you use Firefox. We are also still working on the display issue of NTOC in Firefox2 on Vista. Bug 20669
The release details and discussion can be found in the Babaco discussion page of the usability project wiki. Looking forward to receiving your feedback.
- Naoko

Navigable Table of Contents

Navigable Table of Contents

Inserting link using link dialogue

Inserting link using link dialogue

1 Comment

MediaWiki’s new discussion system in testing on Wikimedia Labs

I’m very excited to announce that LiquidThreads, the next-generation discussion system that I’ve spent the last few months developing for the Wikimedia Foundation, is now in beta testing on liquidthreads.labs.wikimedia.org.

Sample of the LiquidThreads interface

Sample of the LiquidThreads interface

Read the rest of this entry »

, , ,

3 Comments

LocalisationUpdate in testing

Ok, we still need to complete automation of update runs for LocalisationUpdate, but it seems to be working!

It’s not the most glamorous of extensions, but you can see here an updated message (”Den här sidan” where the current deployed message file says “Denna sidan har”)!

– brion

No Comments

LocalisationUpdate and ProofreadPage second try tomorrow; Collection too

Updates are coming back for LocalisationUpdate and ProofreadPage extensions, which we tested then pulled Tuesday due to performance problems.

Roan’s redone LU to store the message updates in serialized files instead of the database, which we can sync locally to web servers and should perform much better; I’ll also do a more gradual test rollout so we can scale it back more gracefully if we have problems again.

ThomasV has fixed up some bad queries in ProofreadPage, and it should be ready to go again; the updated version has much more advanced index support and looks pretty spiffy. :)

And if we’ve got time between other things, I’ll roll out the updated Collection as well — this makes big improvements to the UI for building a multi-page book, and leaves the sidebar much less cluttered when you’re not using it.

– brion

No Comments

Commonist, CommonsHelper fixed

I’ve put in a fix for uploader tools and bots that have been broken since the last update — these bot tools didn’t expect the upload form to change, so they don’t pass in new required fields such as the edit token which was added to the form in the latest update.

Since the edit token isn’t actually required for web uploads (it’s a protection against a class of attacks which, as it happens, can’t forge file uploads) I’ve relaxed the check. I’ve confirmed that it fixes Commonist and have a report that CommonsHelper is also fixed.

Most other bots and tools that were affected are probably also fixed; please test them and let us know if anything’s still broken!

,

4 Comments

LocalisationUpdate deployment delayed

Problem #1 — causes huge increase in CPU load on web servers

Problem #2 — completely kills THE ENTIRE SITE when you DISABLE it because serialized LUDependency objects in the message cache can’t be reinstantiated.

broke

Sigh… Why can’t life be easy :)

Update 2009-09-23 16:30: Roan is starting work on replacing the database storage layer with a flat-file which should be more efficient for our use case.

No Comments

Help us make Commons work better

It’s come to my attention that Commons has had some more problems than we saw on the other wikis… I want to make sure we get that cleaned up!

Andrew has just deployed some more fixes which should eliminate the ‘file already exists on Commons’ bug. It would be a huge help if folks who’ve been reporting problems can go through the lists on the Village Pump and collect just the problems that are still current:

  • Uploads via Commonist — fixed or do we still have problems?
    • (Who’s the best person to contact about debugging or developing fixes for Commonist?)
  • Missing description page immediately after edit — is this still happening?
  • Anything else?

Please add info about confirmed problems or fixes at the Commons Village Pump. Thanks to everybody — your feedback is important; we need to know which bugs are affecting you guys the most!

– brion

Update 2009-09-22 23:49 UTC: I’ve fixed the problem with Commonist by relaxing the edit token check in the form handler (this is safe for uploads via the web, which can’t be injected in a CSRF attack; we still enforce it for upload-by-URL). This probably fixes most of the other client-side upload tools and bots as well — please test!

1 Comment

File renaming enabled for admins

Renaming of uploaded files has been disabled for the last couple of months pending software updates; as of today it is now re-enabled for admins on all Wikimedia sites.

Local admins can now rename files as well as deleting them; this makes it easier to track history for images and other media that need to be moved to another name.

– brion

Update 00:49 UTC 22 Sep: We’ve tracked down a bug which at least on en.wikipedia would cause the redirected old name to stop working for 24 hours. Now fixed; notes for cause and workaround on enwp Village Pump.

1 Comment

Beta edit toolbar disabled temporarily

Due to compatibility problems on Internet Explorer after yesterday’s code update, I’ve temporarily disabled the Usability Initiative’s beta advanced toolbar. If you’ve had it enabled, you’ll just get the regular old edit toolbar until we re-enable it.

Hopefully we should have this resolved within a day or so, and it’ll be back on for all our happy testers!

– brion

Update Sep 18 14:13 UTC: the bug has been fixed, and the toolbar has been reenabled.

2 Comments

Babaco Preview

Screenshot of Navigable Table of Contents

Screenshot of Navigable Table of Contents

As brion’s earlier post stated, the new set of usability features (nickname: Babaco)are integrate into the source code but disabled until these features are stable.  New features include, navigable table of contents and content generation dialogues for links, tables, search and replace.  These features are currently staged on the usability prototype environment, and you are welcome to check them out and encouraged to push them hard to find bugs. :-)

We are also hoping to integrate add-media-wizard created by Michael Dale in Babaco. While API to extend the toolbar is being worked on, media import functionality can be played at the usability sandbox environment. Click the “Edit” tab and click an image gallery icon in the toolbar.  You will see a nice media import function there.  Feel free to create an article and import image from Commons using this neat feature.

Browser compatibility matrix for Babaco is found here. As always, share your thoughts and experience through the usability wiki.  Feedback on Babaco goes this discussion page.

- Naoko

,

5 Comments

Software updates Wednesday morning

I’ve been spending much of the last few work days tidying up an update to our deployed codebase, which has been several weeks behind development for most components.

I’ll want to start deploying this in the morning (Pacific time), so we’ll have most of the day to poke around and fix up any problems on the sites… it’s gonna be fun!

This’ll primarily bring a lot of under-the-hood improvements, which’ll let us start rolling out other fun things over the coming days/weeks including:

For those of you poking at the code, I’ve got the pre-deployment code currently sitting in the wmf-deployment-work branch; this’ll get folded back over wmf-deployment when we’re ready to go.

I believe all custom hacks from the current wmf-deployment branch have been either copied over or generalized and merged via trunk… Note I’ve held back updates on ProofreadPage and OggHandler for the moment, and we won’t deploy the new JS2 code yet until we’ve shaken things down some more.

If there are any *critical* trunk fixes from the last few days (since r55160 trunk branch point) that need to be forward-ported before we start, or any other surprises, let’s make sure we know about it soon. :)

– brion

Update 1:30 UTC Sep 17: Most problems have now been shaken out. Probably a few more minor glitches to go, then we can worry about prepping the fun stuff! :)

9 Comments

PDF Export currently down (fixed)

Our PDF export server is presently down.  It had to be rebooted to organize and route some power cables in our racks.  When it powered back on, it is failing to load all software correctly.  We are working on resolving it, I just wanted to post something here on the blog since it is the first place that many people check when they think some service is broken.

,

No Comments

Improving Wikimedia’s Discussion System

Hi all,

Some of you might have already seen my blog posts about LiquidThreads, Wikimedia’s in-development discussion system.

For those who haven’t, this is a quick primer on what LiquidThreads is, and what it’s going to do for Wikimedia’s communities.

Currently, Wikimedia’s discussion system sucks. Here’s why:

  • It’s not easily usable by the average user. It isn’t obvious how to leave a comment on a talk page, or how to reply to a comment. The indenting we use now is ad-hoc and unsustainable for long discussions.
  • Signatures are done manually and we have to jump on poor unsuspecting newbies who don’t know this (or write bots…)
  • Archiving is done unevenly by bots, which are maintained by users and therefore of very uneven quality. Archives are something of a black hole — they aren’t searchable, easily maintainable or easily accessible. You can’t resurrect an archived discussion easily, nor can you view its history.
  • It’s stored as plain wikitext, which is opaque to any sort of automated process.
  • You can’t move a thread to a different discussion page and preserve its history.
  • There’s no encouragement, mechanism or incentive for quoted, point by point inline replies like we’re all used to with e-mail.
Imagine being a new user and trying to figure out how to add your comment to this.

Imagine being a new user and trying to figure out how to add your comment to this.

Enter LiquidThreads. LiquidThreads is a system that makes MediaWiki’s discussion system behave like a forum or comments thread, while still maintaining the unique refinements that make wikis work. It was originally designed by a Google Summer of Code student, David McCabe, and I’ve been making incremental improvements to make it work for Wikimedia.

Overview of the new LiquidThreads interface

Overview of the new LiquidThreads interface

So, what’s changed?

  • Comments are separated from each other in the wikitext, so there are no more edit conflicts in discussions, and the usability is vastly improved.
  • Instead of indenting, each comment is in its own box, along with its replies. It makes it much easier to follow each post and its replies, and it’s much nicer on the horizontal whitespace. Hopefully, it will be the death of the ‘arbitrary section break’!
  • Each post has its own history page, making it easy to see what’s going on with individual threads without trying to navigate the history of a whole page.
  • It’s easy to move threads between pages, preserving the page history.
  • Discussions  are never ‘archived’. Instead, older discussions fall to the bottom of the page, and eventually they drop off entirely, to hit a new page. If you missed the chance to have your say, just reply to a discussion and it’ll be bumped right up to the top of the page again!
  • Discussions with recent changes are at the top of the page. Discussions that have fallen dormant fall to the bottom. It’s easy to find out what’s happening!
  • You can watch individual threads of a discussion, and even get an email when they’re replied to.
  • It’s easy to link to a discussion, and the links are permanent unless the discussion is deleted. There’s no need to point to an archive or to an old revision ID.

If you’re interested, I’ve put together a test setup for you to play with it.

As always, questions, comments and suggestions are more than welcome, in the comments or elsewhere.

, , ,

8 Comments

First usability release, Acai, is now available.

Screenshot-Editing July 1 Wikipedia

The first usability release, Acai, hit Wikipedia and sister projects this afternoon. The new skin, Vector, and the enhanced toolbar can be turned on from the user preference under “Appearance” and “Editing”. Search result page now has a new layout with less daunting information. Vector is only available for left-to-right languages at a moment due to IE6 incompatibility. However, the enhanced toolbar can be selected from all languages and the new search result page is enabled globally. We could not roll out two features we had planned. First, warning messages for unsaved changes when a user switches away from the edit tab did not work properly thus they are disabled. So please be careful when you switch away from the edit tab. Secondly importing language specific configuration for special characters were not graceful, so we disabled special character function from the toolbar. We are working on the fixes and plan to roll them out as soon as we have stable solutions. The usability project wiki has Vector and the new toolbar as a default, so if you prefer to check them out without changing your preferences it is a good place to visit first. Let us know what you think. We would love to hear from you.

Best,

Naoko

, ,

No Comments

Wikimedia Mobile is Officially Launched

iPhone Version in English

iPhone Version in English

After spending about 6 months in alpha-beta-development-maybe-kind-live mode, we have recently moved Wikipedia Mobile over to a new fast and sexy server. With this new server, we’ve reached the point in development where we can call this baby “launched”!

When I was brought on board at Wikimedia, I was tasked with endowing Wikimedia with a compelling mobile offering. From the beginning, we knew we were going to focus on “fully featured” smart phones. These phones are taking more and more of the market and we believe they will have an easy majority-share in a couple years. The goal is to build for the future.

At the moment, the Mobile site supports iPhone, Kindle, Android, and Palm Pre. And we fully support both English and German. There are other working languages, but they haven’t been fully translated yet. Our goal is to grow slowly and do it really well. We are starting out simple with limited support in order to test the usability and the platform’s stability. So far, things are looking good.

During the beta test period, we’ve served around 10,000,000 pages. You can view the hourly stats here (updated every hour on the hour). And with this new test server, we should be able to do more.

Based off of requests from Google and the Palm Pre folks… and with what just makes sense. We are doing default mobile redirects. That is, if you open a wikipedia link on a supported mobile device, then you get redirected automatically to the mobile gateway. If you click the “View this page on main Wikipedia” then we disable that redirect with a cookie. This way, the 99% of people using mobile devices to read Wikipedia on-the-go have a seemless experience. And, the 1% who like to edit on their mobile device can use their browser to view the main site and do all the fancy things that they like doing. We suspect an initial outcry from the editors that use their mobile devices, but hope that will calm down. We’ve had very good feedback from the 99% and so we can’t forget those folks. If anyone has any suggestions on how to make this easier for the 1% who are editing while mobile, we’d love to hear from you.

If you want live updates about the Mobile site then you can follow WikimediaMobile on Twitter. Also, if you know any Ruby, you can grab the source code via git from Github and helpout! Feel free to contact me via email with any questions.

Also, special thanks to Nic Williams and Ryan Bigg from Mocra for help with the Ruby 1.9 transition and thanks to Yahuda Katz for help with the XML parsing layer and for all his work on the Merb framework.

48 Comments

On templates and programming languages

As many folks have noted, our current templating system works ok for simple things, but doesn’t scale well — even moderately complex conditionals or text-munging will quickly turn your template source into what appears to be line noise…

<includeonly><span style="white-space: nowrap;">{{#if:{{{3|}}}|
{{coord|{{{1|0}}}|{{{2|0}}}|{{{3|0}}}|{{{4|N}}}|{{{5|0}}}|{{{6|0}}}|{{{7|0}}}|{{{8|E}}}|{{{9|type:other}}}|format={{{format|dms}}}|display={{#if:{{{title|}}}|inline,title|inline}} }}| {{#if:{{{2|}}}|
{{coord|{{{1|0}}}|{{{2|0}}}|{{{4|N}}}|{{{5|0}}}|{{{6|0}}}|{{{8|E}}}|{{{9|type:other}}}|format={{{format|dms}}}|display={{#if:{{{title|}}}|inline,title|inline}}}}| {{#if:{{{4|}}}|
{{coord|{{{1|0}}}|{{{4|N}}}|{{{5|0}}}|{{{8|E}}}|{{{9|type:other}}}|format={{{format|dec}}}|display={{#if:{{{title|}}}|inline,title|inline}}}}| {{#if:{{{1|}}}|
{{coord|{{{1|0}}}|{{{5|0}}}|{{{9|type:other}}}|format={{{format|dec}}}|display={{#if:{{{title|}}}|inline,title|inline}}}}}}}}}}}}</span></includeonly><noinclude>
{{pp-template|small=yes}}
{{documentation}}
</noinclude>

And we all thought Perl was bad!  ;)

Lua

There’s been talk of Lua as an embedded templating language for a while, and there’s even an extension implementation.

One advantage of Lua over other languages is that its implementation is optimized for use as an embedded language, and it looks kind of pretty.

An inherent disadvantage is that it’s a fairly rarely-used language, so still requires special learning on potential template programmers’ part.

An implementation disadvantage is that it currently is dependent on an external Lua binary installation — something that probably won’t be present on third-party installs, meaning Lua templates couldn’t be easily copied to non-Wikimedia wikis.

There are perhaps three primary alternative contenders that don’t involve making up our own scripting language (something I’d dearly like to avoid):

PHP

  • Advantage: Lots of webbish people have some experience with PHP or can easily find references.
  • Advantage: we’re pretty much guaranteed to have a PHP interpreter available.  :)
  • Disadvantage: PHP is difficult to lock down for secure execution.

JavaScript

  • Advantage: Even more folks have been exposed to JavaScript programming, including Wikipedia power-users.
  • Disadvantage: Server-side interpreter not guaranteed to be present. Like Lua, would either restrict our portability or would require an interpreter reimplementation. :P

Python

  • Advantage: A Python interpreter will be present on most web servers, though not necessarily all. (Windows-based servers especially.)
  • Wash: Python is probably better known than Lua, but not as well as PHP or JS.
  • Disadvantage: Like PHP, Python is difficult to lock down securely.

Any thoughts? Does anybody happen to have a PHP implementation of a Lua or JavaScript interpreter?  ;)

– brion

Update:

Hampton reminds me that Ruby has some sandboxing features and may also be a contender.

, , ,

11 Comments

Blog Downtime

I am sure that many folks noticed that on the morning of 2009-06-26, techblog.wikimedia.org and blog.wikimedia.org went down.  It turns out that some of the parts of our Wordpress installations were compromised.  I do not want to get in to a direct show and tell of what they did, but hopefully we have hardened the installation to the point that it will not occur again.

This is why the blogs exist on their own server, so when things like this happen we can minimize the impact.  The blogs are both up and running now, along with the other services that were affected.  All but techblog was back online before Friday was over, techblog lagged behind until today.  (As techblog was the point of exploit, we got everything else back up first.)  Other affected services were the Open Conference Systems site for Wikimania 2009, as well as our survey software.  Both of those were back online ASAP after the incident and the rest followed after.

Of course, it was hard to get this information out to folks when the blogs were down!  It goes to show how easily using the blogs to get info out has been, since without it we had to scramble to get the information out of other channels.

Thanks to everyone who assisted in the restoration, and also thanks to everyone for their patience while the system was fixed.

No Comments

IRC server refresh for irc.wikimedia.org

In a couple of minutes, I will be shutting down irc.wikimedia.org and the rc bot to perform a much needed upgrade of the irc server.
We have been experiencing problems with the current server, with IRC feeds not showing up, and the goal is to remedy these problems, as well as incorporating new functionality.

The documentation for the new new server will show up shortly thereafter on  Wikitech.

It is excpected that there will be a little downtime during which users will not be able to connect to irc.wikimedia.org. The upgrade should take about 10 minutes, and there is a rollback plan in case something goes wonky.

1 Comment

The Wikipedia Usability Initiative is still hiring.

The Wikipedia Usability Initiative has extended the application deadline for the Software Developer position till May 30th. We are recruiting two candidates for this position. Both local applicants to the San Francisco Bay Area and remote applicants are encouraged to apply. Please help spread the word.

http://wikimediafoundation.org/wiki/Job_openings/Software_Developer_(project)

Naoko Komura
Wikipedia Usability Initiative

, ,

No Comments

Skin & JS cleanup and jQuery

Just a heads-up –

Michael Dale is working on some cleanup of how the various JavaScript bits are loaded by the skins to centralize some of the currently horridly spread-out code and make it easier to integrate in a centralized loader so we can serve more JS together in a single compressed request.

Unless there’s a strong objection I’d be very happy for this to also include loading up the jQuery core library as a standard component.

The minified jQuery core is 19k gzipped, and can simplify other JS code significantly so we can likely chop down wikibits.js, mwsuggest.js, and the site-customized Common.js files by a large margin for a net savings.

If you’ve done browser-side JavaScript development without jQuery and wanted to kill yourself, I highly recommend you try jQuery — it’s sooooo nice. :)

21 Comments

Wikimedia Deutschland offering short term contracts

Wikimedia Deutschland is offering contracts for a couple of projects we feel are important. If you are interested in earning some money by helping Wikimedia to improve our Wikis, have a look at these projects:

If you would like to help with any of the above, please contact User:Duesentrieb aka daniel.kinzler AT wikimedia.de and provide the following information:

  • Your real name and country of residence
  • How you plan to go about implementing the desired function
  • Any experience working with MediaWiki
  • How many working hours you would spend on it, and how much you ask for it
  • In what time frame you would be able to do the job

Thanks for helping us make the web a better place!

1 Comment

Code updates going live to Wikimedia sites

After a few weeks of bug fixes, we’ve caught up with MediaWiki development code review and I’m pushing out an update to the live sites. This fixes a lot of little bugs, and hopefully doesn’t cause introduce too many new ones. :)

As usual in addition to lots of offline and individual testing among our staff and volunteer developers, we’ve done a shakedown on test.wikipedia.org — and as usual we can fully expect a few more issues to have cropped up that weren’t already found.

Don’t be alarmed if you do find a problem; just let us know at our bug tracker or on the tech IRC channels (#wikimedia-tech on Freenode).

We should be resuming our weekly update schedule soon, and will continue to improve our pre-update staging and shakedown testing to keep disruption to a minimum and awesome improvements to a maximum.

For those of you reading via the Planet aggregator, I’d also like to announce that we’ve started a blog for Wikimedia tech activity & MediaWiki development, in part because I want to make sure community members can easily follow what we’re working on and give feedback before we push things out.

I’d very much like to make sure that we’ve got regular contacts among the various project communities who can help coordinate with us on features, bugs, and general thoughts which might affect some projects distinctly from others.

– brion vibber (brion @ wikimedia.org)
CTO, Wikimedia Foundation
San Francisco

1 Comment