In SVN: wml2po

Some say it's better late than never, I say it's something I should have done a little bit earlier. It took some time before I managed to get on the right track and find a solution that everyone seemed happy with. Hopefully, this is it. The script, wml2po is now in the tor svn repository.

Almost ready: wml2po

The script, wml2po, is almost ready. It assumes that the contents of website/ is split into wml/ and po/. The resulting po files are now being put where they belong, i.e.:

• website/wml/en/index.wml to website/po/$language/index.po


• website/wml/press/en/index.wml to website/po/press/$language/index.po

Also, the hash of each po file is generated before and after the file has been updated. The only thing left is to make sure that script has a proper lockfile.

My progress so far

I just sent an email to the tor-gsoc list asking if it is possible to split the contents of website/ into po/ and wml/. Doing so will make things a whole lot easier when it comes to figuring out where to put the resulting po files.

I also sent an email to my mentor telling him about my progress so far. What I wrote is pretty much the same as I have previously written in this blog:

Over the past few weeks I have written a few different scripts. While they have worked, they have not been ideal solutions to the problem I am trying to solve. If you read the blog post from yesterday, July 2nd, you'll see that I've gotten useful pointers from Peter and that
is what I have been working on today.

What's left to do is:

For wml2po:

• Make sure that the wml and po files are put where they belong.


• Generate hash values for each file before and after converting. Compare these two values and revert if they are the same (i.e. nothing has changed).

For po2wml:

• The code will be pretty much the same as for wml2po and I don't think writing this script will be a problem.

Converting from wml to po, including translations:

• I have already converted the wml files in the repository (including their translations) to po files. I'm just not sure where I should upload these. What do you think?


• I also think that uploading those po files won't do much good until we have a script that can keep them updated automatically.

According to my timeline I should be done with the coding by the end of next week, and then start to write documentation (that is, document the code properly and write a readme). I am confident that I'll be able to stick to that plan. Let me know if you have any questions.

A few bumps in the road

Me and my code have been hitting a few bumps in the road over the past couple of weeks. Either because the solution I came to simply wasn't the best solution out there or because I misunderstood something. Peter Palfrader has been really nice and given me a few pointers, and I am confident that I am on the right track.

So, the new approach will be to use a proper lockfile (and not svn lock), comparing files with hash values, converting all of the wml files, reverting the ones that haven't changed since last time and committing the ones that have.

There is also one more thing on the todo-list, other than writing a script that will keep all the files updated automatically, and that is to convert all the wml files to po files. This can be done in just a few minutes (given that all the files can be converted without problems). I figured I'll upload these to the repository when:

1. The fancy, automatic wml-to-po-and-back-script is done.
2. I know where to put the po files.

I also plan on writing a longer progress report tomorrow and email this to my mentor, Jacob Appelbaum, as well as post it here.

I spoke too soon

Yesterday wasn't much fun. I was in bed with the flu, and so Monday passed and no code was uploaded to the repository. I am feeling a bit better today and have had a look at the code I wrote last week. I also noticed that my todo list was missing an important item. Item number 4, from the previous post, says:

I need to make sure that svn lock is used at the right places, and that the script also adds and commits the files before it's finished.

In addition to that, I also need to think about the small details when it comes to locking and unlocking. What should happen if the lock can't be aquired? What should happen when svn up results in conflicts or the commit doesn't work? To me, exiting and letting the next invocation handle it seems like a simple way of doing it.

Hopefully tomorrow will be a more productive day.

New and improved code

Today was spent rewriting old code into something much better. The code that will take new and updated wml files and convert/update the po files are done. I will be leaving for Oslo tomorrow, but I will continue working on Monday when I'm back in London. What remains is:

1. The script needs to know what languages are supported by Pootle. For now, this is done by reading a simple text file. In the future, this could be done by getting a list of directories in /var/lib/pootle/pootle (given that this directory still exists on the server where Pootle is set up).

2. The script needs to check svn up twice. Once to figure out which wml files are added/updated, and once to figure out which po files are updated. To deal with this, I plan on introducing a file, svnup.tmp, that the script will generate, read from and then delete.

3. I need to add the code that will take the updated po files and convert them back to wml. The code will not be much different from the code I wrote for the svn hooks, so this should be quick and easy.

4. I need to make sure that svn lock is used at the right places, and that the script also adds and commits the files before it's finished.

Thoroughly documented and fully working code will, hopefully, be committed to the tor svn repository on Monday.

If at first you don't succeed

I decided to look into po4a and its configuration file, hoping that it would be a better solution than svn hooks. Turns out it was a dead end. Using the configuration file will only help keep the translated documents updated (for example, the Norwegian version of gsoc.wml), but it doesn't do much for the pot and po files.

I have, however, received good feedback on the email I sent to the tor-gsoc mailing list last week, asking for suggestions and ideas regarding my solution with svn hooks. One of the suggestions was to have a script that one can run any number of times, and if the input hasn't changed it won't do anything.

So, my new and improved plan is to take the hooks that I have, clean up the code and, instead of checking for new files in a revision/transaction using svnlook, checkout/update a working copy of the HEAD and find out what files I need to update from svn up.

Some questions have been raised as to what should be done with Pootle. At the moment, Pootle needs to be restarted to learn about new files. The good news is that this will be improved in the next version, where one can tell Pootle to rescan for files from the admin interface. A possible, but not pretty, solution could be to have a cron job that runs the script and restarts Pootle once a day.