In order to find out what's actually happening to it I've used the tool called
bleak_house, which is very good to examine how your object space is evolving.
Istallation is fairly simple:
gem install bleak_house
And then just run you application using the ruby-bleak-house instead of a regular ruby.
Basically what bleak_house does is it dumps an object space picture to a disk when you press Ctrl-C, or send a USR2 signal to the process.
So I used the second method from crontab to get an image of the process every minute.
Then you can run bleak command to analyze the dump and see what has changed.
In our case it happened that logs were not flushed (sic!), but since we are currently running our application with debugging logs enabled - it led to very fast process growth.