Dev Blog AF83

L'architecture.

Les bots

Il ne reste qu'à exploiter les capacités de Rails et de XMPP afin de développer les bots qui vont effectuer les traitements.

Exemple :

Ces lignes permettent à un script Ruby de charger l'environnement Rails du projet. Le bot est alors capable d'attaquer notre modèle de données via ActiveRecord.

Ce gem nous permet de communiquer vers le compte Jabber du bot. Il peut donc dépiler les objets qui lui sont destiné et les désérialiser pour les traiter.

Ce gem permet de gérer le bot en tant que service (stop/start/restart).

Pour sérialiser/désérialiser vos objets Ruby / Rails.

Ce gem sert à intercepter une coupure de la connexion vers MySQL. Il surcharge ActiveRecord afin de renégocier une connexion.

Avec ces outils nous avons la capacité de créer des services exploitant une file d'attente Jabber et fonctionnant en parallèle. Ainsi, si la charge vient à augmenter, les messages en attente de traitement ne seront pas perdus, puisqu'en attente dans les comptes Jabber stockés par ejabberd. De plus il est possible de multiplier un même bot en exploitant les ressources du protocole Jabber, chaque bot écoutant sur sa propre ressource (bot@monserveurjabber.com/1, bot@monserveurjabber.com/2, ...).

Résumé

Pour Noumba, nous avons développé un projet en Rails, le Hub, qui gère la file d'attente Jabber, et communique en REST avec le frontal Noumba. Si cette architecture est sur-dimensionnée pour votre projet, nul besoin d'un backend. Votre site et des bots suffisent amplement. De même quelques comptes Gmail suffisent si vos ne souhaitez pas déployer votre propre serveur Jabber.

Si un backend en Rails s'avère nécessaire il n'est pas conseillé de dupliquer les modèles de votre site principal vers celui-ci. Cela fonctionne mais le principe DRY est de fait supprimé. Cependant un outil tel que Piston permet de temporiser cette affirmation.

Les autres

Après divers essais Twitter a fini par développer son propre serveur de file qui exploite memcached, starling

Un développeur de Seesmic indique utiliser ActiveMQ et RabbitMQ mais semble vouloir migrer vers une solution XMPP : scaling-questions-and-issues

Il existe un grand nombre d'alternatives, on peut citer :

• Beanstalkd , about-this-blog-beanstalk-messaging-queue

• Sparrow

• AP4R

Les deux premiers utilisent le twisted like eventmachine

Bémol

Une telle architecture implique la gestion d'un serveur Jabber, ce qui peut s'avérer une tâche plus complexe et lourde qu'un réel MoM dédié. De plus la stabilité de la bibliothèque xmpp4r ainsi que le plugin ActionMessenger est à surveiller de près. Pour ce dernier, il a été nécessaire de le patcher afin qu'il puisse supporter plusieurs instances Rails.

Avenir

XMPP est un protocole standard et ouvert très répandu ce qui en fait un candidat idéal si l'on souhaite une architecture pérenne et évolutive. L'architecture décentralisée de Jabber et ses capacités à se connecter à des services externes tel que OpenID (xmppid.net) ouvre la porte à une multitude de possibilités.

Ses nombreuses fonctionnalités dédiés au chat (room, pub/sub, voIP, ...) sont toutes indiquées pour des sites communautaires et sociaux à tel point que le projet DiSo souhaite l'utiliser en son coeur, mais ceci est un autre sujet.

We have a web application coded with Rails framework and we want it to be compatible with as many instant messaging networks/protocols as possible.

The fist protocol we supported is XMPP/Jabber. This is an open standard, thus librairies are available. We used xmmp4r-simple to develop our interface, and it works fine.

We had to have a look at what is possible to do for others protocol such as MSN, AIM, Yahoo, and ICQ which are proprietary. The first solution we worked on is based on the gateways provided by jabber. We worked on it because it is very transparent for the client (our web-application), so, developing an interface with others IM would have become a minor task.

Jabberd, a historical yet still used jabber server is included in Debian (our hoster's favorite OS) and gateways are provided too (packages : jabber jabber-msn jabber-jit jabber-yahoo jabber-aim). Everything seems to be one-click away from a perfect working solution.

Only : it doesn't work. First, the installation is not as easy as it seems to be. We had to dig for documentation in the deep web and it had been hard to set up one gateway. During this work, we discovered that those gateways are no longer maintained. Two of them don't follow the protocol evolution and simply do not work. The two others still working, but for how long ?

The gateways provided as jabberd modules are not a solution. So, we decided to find others gateways. We found more recent ones but they bring others issues : some of them are not maintained and their resource-consumption (they're python powered) is two high according to experience return from the jabber community.

So, we decided to change our solution.

Pidgin (formally known as Gaim) is a famous multi-protocol instant-messaging client. Their developers have done a strict layer-separation : libpurple provide support for the IM protocols and Pidgin is just a GUI. Pidgin is a long established and successful project and we can count on the active community and developers to maintain it for a long time. The last remaining problem is that libpurple is coded in C and we are using ruby. It's solved by a wrapper called ruburple but this project is very young... wait and see...

In conclusion, we can say that jabber is a powerful and developer-friendly instant messaging protocol but its current gateways are not a solution for professional use, where uptime and reliability are critical.

We will keep using xmpp4r-simple for jabber support and will use libpurple for the others IM system. We have not tried to code anything with ruburple yet, once we done it : we will provide our feelings here. As soon as we have some interesting code we will be sure to share it with the community.

We thank the french jabber community whose active members provided us with good advice.

Of Lately we have been churning more and more Ruby on Rails code for our web applications. Its source-code is becoming longer and longer. It was time for us to think about formalizing the documentation process.

After a short search, we found rdoc and it seems to suit our needs. It has good source-analysing basis and interesting features such as diagram generation. The problem came from rdoc youngness which led to two issues :

• the lack of documentation, the only relevant document is the rdoc's README file
• In fact, there are no standards nor conventions on tags to describe a parameter, an instance variable, the author of the code or anything else

We had to understand that this

is not a tag. It's just a syntax for something to be parsed as a definition list.

So we decided to make our own convention, trying to be as near as possible to existing documentation conventions while banking on the emergence of a hypothetical future standard. So we looked at the Ruby source-code, seeking for its developpers conventions. Guess what... There are no comments in Ruby code nor in rdoc's one (or very few, almost none are structured).

So... we were forced to set out to do our own convention. As we do a lot of PHP code, we decided to use the inspiration of the JavaDoc inspired phpDocumentor syntax and PhpDoc tag names. We added the :: to keep compatibility with rdoc and we added home-made syntax rules such as the ## to spot the begining of a comment readable with our new convention. And the result is...

Strict conventions on documentation are a necessity today. It makes running automated systems on the code much easier, the lack of such a convention is a gaping hole in the Ruby On Rails arena. And no, the code is not the documentation, neither are Unit Tests.

What we tried was to get the minimal set of "tags" that are necessary for an API style documentation which makes writing the code in a distributed fashion much easier. While keeping in mind how important it is to stick to the existing so that we can re-use tools that are already there.

Now, we can start commenting soundly... hoping others will jump on the wagon and we can start working on a general solution.

How to run several instances of a rails app with a single code base?

What is the problem?

Sometimes you want to run several instances of the same application on your server. How to easily do this in rails?

The solution

Rails offers us the environments, allowing to run an application under different status: development, test or production environment.

We just have to create in the database.yml a specific database connection for each new environment:

prod1:
  adapter: mysql
  database: prod_one
  username: root
  password: password

and for another instance

prod2:
  adapter: mysql
  database: prod_two
  username: root
  password: password

and to create a specific configuration file 'prod1.yml' in the directory config/environments/.

After this, you just generate the database schema with the following command:

rake db:migrate RAILS_ENV=prod1

And run the server on a specified port:

./script/server -p 4001 -e prod1

Et voilà! an instance running on the specified port (4000).

If you want another one (let's say prod2).

Let's create the database instance... rake db:migrate RAILS_ENV=prod2

./script/server -p 4002 -e prod2 ... and another instance of the same code base is now running on port 4002.

Bliss.

Ruby on Rails is a great framework, but would you believe it .. it has no built-in support for localized error messages for model level validation.

The problem is that you can internationalize the error messages but not the field name. Which makes for some very weird behavior.

Here is a bit of code to solve the problem. You have to install localization_generator before.

How to have clean internationalized validation error messages in Ruby on Rails?

in the file app/contollers/application.rb

And that's it! the magic is done with the l() function from localization_generator