Upcoming Globalize feature: an alternative way of storing model translations

posted: January 31st, 2007 · by: Sven

in: Globalization, Programming · tagged as: , , , , ·  30 comments »

Saimon Moore, one of the Globalize committers, pointed me to some terrific new Globalize functionality that most likely will make it into the next (for-1.2 that is) release of Globalize (which is expected to arrive pretty soon).

Saimon has described said feature himself in a blog article: Alternative implementation of Globalize Model Translations. But there’s a lot of code to read over there so I thought I’d sum things up a bit for you.

As the title of Saimons blog post mentions this feature deals with a different implementation of model translations.

With Globalize you’d say, for example:

def Page < ActiveRecord::Base
  translates :title
end

Now, traditionally, your database schema would – regardless of how many languages you’d have to deal with in your application – include a pages table with one column named title, right? This column would hold the base_language translations of titles. All other translations would go into a separate table named globalize_translations – a central storage table for all translations of all your “translates-ed” attributes.

For example, the page title value “I18n the easy way” would be stored directly in the pages table (assuming that English is the base_language of our app). The German translation “I18n leicht gemacht” of this title would go into the globalize_translations table however.

This approach has some obvious advantages like that the translation of your application is entirely separated from the rest of your application/data. But there are some serious drawbacks also, most importantly being less flexible due to the necessity to JOIN in the globalize_translations table. For example, Globalize hasn’t been able to support the standard :include and :selectbehaviour of ActiveRecord (which in turn breaks, ahem, ... disables other ActiveRecord default stuff like has_many :through).

Enter Saimons Alternative Model Translations …

Saimons approach turns things around from a traditional Globalize perspective. His alternative implementation of ActiveRecord::Base#translates:

  • requires you to store your translations directly in the model tables on the one hand but
  • allows you to leverage all the ActiveRecord::Base::find functionality you’re used to without any restrictions on the other hand (such as not being able to use :include and :select in the traditional approach)

As Saimon told me you’ll be able to switch this behaviour on by putting Globalize::DbTranslate.keep_translations_in_model = true in your environment.rb. But you’ll also be able to overwrite this per model, in class scope: self.keep_translations_in_model = true.

To store the translation data directly in the model table you have to have a column per supported language per translated attribute. That is, for the title attribute above, we’d have three db columns when there are three languages supported:

class CreatePages < ActiveRecord::Migration
  def self.up
    create_table :pages do |t|
      t.column :title, :string
      t.column :title_de, :string
      t.column :title_fr, :string
    end
  end
end

It’s been noted that this might get a bit cumbersome as soon as you’ve more than very few attributes to translate and very few languages to support. (On the other hand, one might come up with some automation support based on ActiveRecord::Migrations to manage this kind of stuff.) Also, what if you want to add new languages in an entirely dynamic way, e.g. through a web interface? Well, of course, as always … peoples mileages vary and there’s no silverbullet to whatever job you’re trying to get done.

What Saimons approach is buying you is the removal of a beforehand necessary JOIN. Depending on your situation – that’s probably quite a lot.

Also, as a side-effect of this all the translations will be pre-loaded with your models, so switching the locale won’t result in an additional database hit.

PS: To be honest, I’m writing this after only having read Saimons article, the code in the Globalize SVN etc. I haven’t played around with this myself yet.

I’d love to see your comments though!

Leave a comment

30 Comments

  1. Mat Schaffer said March 9th, 2007 at 04:13 PM  

    Since Saimon’s comments are closed I thought I’d comment here. He mentions the use of a rake task to do auto migrations. What about a custom generator instead? I think I’d be pretty happy with something like:

    ./script/generate translation post title body en ja

    This might bring up an issue of how to differentiate column names from language codes. As a solution, why not just use full locales?

    ./script/generate translation page title body en-US ja-JP

    I think they’d be easier to parse out. Additionally I think users might be more comfortable with this since they’ve likely used Locale.set many times.

    Great work, otherwise. Giving the users a choice is really key. Well played! :)

  2. Sven said March 10th, 2007 at 03:58 AM  

    Hi Mat,

    Saimon already has checked something in. You might want to have a look at: http://svn.globalize-rails.org/svn/globalize/branches/saimon/for-1.2/features/

    What this does is (quoting Saimon from the Globalize dev list):

    • Modify the ruby source for the model to indicate it’s using the internal storage system
    • Generates the db migration for all model using the internal storage system (via existing generator)
    • Executes this db migration
    • Migrates the translation data from the external storage system to the internal system (with option to delete old translations).
    • Asks user for permission before performing each stage
    • Logging to a specific log file

    I haven’t tried that myself though yet. Please, if you try it out, provide your feedback on the Globalize users mailinglist, if possible. Thanks! :-)

  3. Thomas said May 5th, 2007 at 03:17 PM  

    Hi Sven,

    Wwhile looking for a tool to i10n-ize a new project i have come across Globalize (obviously). I thank all of the developers that have made this nice plugin open source.

    First of all, i like the localize part of dates, numbers…

    However i think that the implementation of the db content could need rethinking. Even with Simons alternative model translation, in my opinion there is a cleaner way:

    Each table with multilingual fields (e.g. ‘product’) has a related translation table (belongsto :product) with productid and language_id as index as well as all the multilingual fields.

    This would enable completely transparent model object translation, keep the base table clean (no …yxzlang fields) and ease addition of new languages. The model could then get and merge the translations unless it operates in the base language.

    One note on view tanslation:

    It would be nice to have an alternate possibility to globalize text like <%= 'some very long text...'.t(:my_key) %> This would prevent huge trkey’s in the globalizetranslations table and - if consequently used - enable changing the base locale. By careful key name selection the developer can even give some context information for translators, e.g. ‘…’.t(:frontpagelegalnotice).

    One step further, changes in the base language (i.e. ‘this text has changed’.t(:akey)) could set a dirtyflag in the globalize_translations table whenever the base language translation differs from the text hardcoded in the view. To keep DB access at a minimum, this should be an optional thing. Maybe someone comes up with a parser utility that will do this work.

    Unfortunately i am too new to Ruby and Rails to be able to implement changes like this. Still i have enough years of experience to contribute oh the conceptual side. I believe that implementing these concepts would give Globalize a huge usability boost.

  4. Sven said May 8th, 2007 at 07:51 PM  

    Hey Thomas,

    those are very interesting suggestions!

    I strongly encourage you to join the Globalize users mailinglist and re-post your suggestions over there, as this blog is not the best medium for exhaustive discussion.

    Some of what you suggest is actually discussed right now amongst the developers. E.g., there are some ongoing efforts of refactoring the two distinct viewtranslation storage mechanisms into one.

    Also: can you expand on what you think would be the advantage of many *_translations tables as opposed to one big table to store the translations?

    Anyways: thanks for your suggestions!

  5. Martin said November 7th, 2007 at 11:37 AM  

    Hey Thomas

    We have been looking at translation plug-ins for some time.

    It ended up in us creating our own here at White Wall Web. Its is completely transparent - and intrestingly enough incorporates everything you have mentioned. See you comment post was over 5 months ago though. Have you found any similiar plugins yet?

    We have using been using our in-house plug in in two international applications and has been a dream working with it - as I mentioned, its completely transparent - so you go about cooking your rails app as you would in any other single language application - everything runs off your language key in the param hash, If no language for the model attribute, it will default to your base language, which could be set to any language off course (on setup). Translations are done by a web interface per model.

    I am currently writing a blog post comparing all the translators out there (including our beta version) so keep an eye on our office blog http://whitewallweb.com.

  6. Pwic said November 10th, 2007 at 11:39 AM  

    This alternative way sucks. This is anti-polymorphic and inheritated from a very bad db design. So let’s say you have 3 columns to translate (do you ever only translate a title ??) into many languages. Until how much columns will you blow your table ?

  7. Sven said November 10th, 2007 at 01:24 PM  

    Hey Martin,

    thanks for the headsup!

    I’d love to have a look at your solution, but unfortunately when I access http://whitewallweb.com it’s bouncing me back and forth between http://whitewallweb.com/?page=home and http://whitewallweb.com/?page=home?page=content&id=1 forever (until Safari kills with an error message).

  8. Sven said November 10th, 2007 at 01:29 PM  

    Hey Pwic,

    saying that the alternative sucks is just harsh.

    The technique is well known as “database denormalization” and it has it’s traditional place in programming to solve issues when they can’t easily be solved otherwise.

    Of course it’s no silverbullet and won’t make sense for many applications. But that’s true for probably every other solution, too.

  9. Fernando said November 17th, 2007 at 01:48 AM  

    Hello, I’ve been trying to decide which tool is the best for me and right before reading this post I thought Globalize was it, but before I make that decision I want to know if you could make it clearer on what exactly you mean by “which in turn breaks, ahem, … disables other ActiveRecord default stuff like has_many :through” ??… Does it mean I can’t use has_many :through at all in my whole project if I use Globalize in the “traditional” way? OR Does it mean I can’t use has_many :through only in the models I’m translating data from?

    Thanks for your reply

  10. Pwic said November 17th, 2007 at 10:36 AM  

    Hi Sven,

    I’m really sorry if I offensed you. However you solution is “just a trick” which is really heretic for a database designer. I have to admin I also don’t like the how the globalizetranslations is designed, the rails polymorphic convention is not followed with column such as ‘tablename’ and ‘itemid’. Storing X times the same trkey is dirty too…

    I really like other features of Globalized, so instead of writing yet another l8n plugin, I’ll submit soon a fork with another modeling for model and view translations.

  11. Sven said November 18th, 2007 at 08:38 PM  

    Hi Fernando!

    Offhand, I think it’s only the case for those models that have that translates :some_field directive added to it. As far as I remember that directive mixes in Globalize’s find method replacements which overload the standard ActiveRecord finders.

    You might want to have a look at the code yourself though (or probably just try it out): http://svn.globalize-rails.org/svn/globalize/trunk/lib/globalize/localization/db_translate.rb

    Let me know how things work out for you.

  12. Sven said November 18th, 2007 at 08:44 PM  

    Hey Pwic,

    I don’t feel offended at all :)

    At the end of the day it’s not even my code, but Saimons. Heh! But I really think that in a world where you’re forced to work around ActiveRecord limitations from a plugin there’s probably no single perfect solution. I totally agree with you that the database scheme used here is not really nice. But it works and actually provides a useable solution in some cases.

    That said, Globalize can definitely use improvements. If you’re going to put some time into this, may I recommend to join the Globalize dev team instead of forking the whole thing? Saimon’s approach already materialized the idea of “storage engines” in Globalize and this will be further expanded in the future anyways. So if you have some thoughts about how this could be done all better and even want to implement it, just drop me a mail and I’ll get you into the Globalize dev team.

  13. Sally, translator said November 27th, 2007 at 03:29 PM  

    Even with your examples there are too much codes for me in your post. Actually, translation plug-ins are an open question., because automatic translation causes a lot of semantic mistakes.

  14. Martijn said December 17th, 2007 at 12:34 PM  

    With respect to Pwic’s remark that “Storing X times the same trkey is dirty too…”

    I couldn’t agree more. Especially when you have long texts you want to translate.

  15. Zach said December 31st, 2007 at 06:43 PM  

    What are thoughts on reworking this to use a serialized hash? So you only have one field (“translations” of type TEXT) and its content is something like:

    :title: :en_gb: Colour :en_us: Color :description: :en_gb: This is a Colour :en_us: This is a Color

    One field for all translatable fields/countries? What are the performance/scalability implications vs the denormalization method?

    thanks.

  16. Zach said December 31st, 2007 at 06:44 PM  

    (post butchered that, needless to say that was a yaml file for the hash which was two levels deep (field to translate, then country)

  17. Sven said January 1st, 2008 at 12:59 PM  

    Hi Zach,

    actually there are some things in the oven to get a patch to Rails core accepted that will rely on yaml files for view translations and stuff like that. Once this is done, Globalize will most probably move to a more modular approach where you can choose from different storage engines for translations.

  18. Rolf said May 20th, 2010 at 05:47 PM  

    Thanks Sven ! I’ll get back to post some comments after I try it (I’m a bit late though !!)

  19. Paul said July 29th, 2010 at 11:15 AM  

    Yep ! Good idea ! I will try it too !

  20. finance said February 6th, 2011 at 03:14 PM  

    I believe I understand what your saying and in essence I think I agree. I would like to hear more about what your saying here thought. I’ll be back.

  21. QQQ said February 7th, 2011 at 06:32 PM  

    Finally we kissed and the passion scale went sky high and I knew I was onto a good thing - sex was a certainty free porn videos. She never hesitated when I began to fondle her breasts and she willingly exposed them for me mobile porn. They were firm and I suspected a breast enhancement but said nothing - they still felt good and I was enjoying them and gradually working my way further south free porn tube. She was a step ahead of me and before I could completely undress her she moved on me atk hairy and I was suddenly having my pants pulled down and I was enjoying one of he best cock sucking hairy pussy experiences I had ever had. ABB728019394

  22. Yeng2 said February 24th, 2011 at 06:47 AM  

    While we abide to accumulate angle on the Iraqi war and the political leaders who led us there, pass4sure 70-443 there continues to be rhetoric. pass4sure 70-519 From those with brave political bias. While in hindsight there was an intelligence debacle, pass4sure 70-529 the aforementioned adulterated advice was initially advised by Bill Clinton with his consecutive approval of the invasion. pass4sure 70-541

  23. David Wilson said March 8th, 2011 at 05:29 PM  

    A new hybrid method of storing data is that of identifying point clouds. In vector data, the additional data contains attributes of the feature and the simplest model is to assume the earth is a perfect sphere which helps to understand the complete circle - ereader comparison

  24. chat said March 31st, 2011 at 08:09 PM  

    The following cleaned up the issue:

    Dependencies.loadoncepaths -= Dependencies.loadoncepaths.select{|path| \ path =~ %r(^#{File.dirname(FILE)}) }

  25. Daniel said April 5th, 2011 at 09:45 AM  

    Great feature. I’m currently on my way to a a tech conference and a wellness kurzurlaub and I’m gone check back on the different features here.

  26. jamesk said April 10th, 2011 at 07:29 AM  

    Well I definitely liked reading it. This information provided by you is very constructive for correct planning. blendtec total blender | delonghi toaster oven | breville toaster oven

  27. side sleeper pillow said April 22nd, 2011 at 07:03 AM  

    Nicely written article, Knowledgeable and informative post. I’m really glad I came my way along your site. Keep posting, I really like the whole topic. Thanks for sharing.

  28. Okey oyunu said May 12th, 2011 at 04:12 PM  

    Thanks for this article. Tüm dünya artik okey oyunu oynuyor. Yillardir bir çok oyun programi olmasina ragmen, içlerinden en güzeli olarak nitelendirebilecegimiz tek bir site göze çarpmaktadir. Diger tüm okey oyunu programlarinin aksine ücretsiz olmasi ve 3 boyutlu olarak hizmet vermesi mükemmel bir gelismedir. Sizlerde www.okey-oyunu.com adresinden bu essiz okey oyununu indirebilirsiniz. Kullanimi çok basit ve Türkçe dil seçenegi ile kolaylikla oyuna baslayabilirsiniz. Ister kendi ülkenizden, isterseniz dünyanin tüm farkli bölgelerinden dilediginiz oyun odalarini seçerek, oyuna hemen baslayabilirsiniz. Okey oyunu oynamak için artik arkadas bile aramaniza gerek kalmadan, bilgisayarinizdan 100 binlerce üye ile online olarak okey oyununu oynamanin zevkine varabilirsiniz.

  29. porno said May 22nd, 2011 at 01:32 PM  

    I do agree with all of the ideas you have presented in your post. They’re really convincing and will definitely work. Still, the posts are too short for newbies. Could you please extend them a bit from next time? Thanks for the post.

  30. porno said May 22nd, 2011 at 02:17 PM  

    good comment. thanks you friends.

    I’ve surfed the net more than three hours today, however, I haven’t found such useful information. Thanks a lot, it is really useful to me

Sorry, comments are closed for this article.

artweb design
Sven Fuchs
Grünberger Str. 65
10245 Berlin, Germany


http://www.artweb-design.de

Fon +49 (30) 47 98 69 96
Fax +49 (30) 47 98 69 97