How I got stung by the Twitter gem and thread safety
If you use sferik's Twitter gem with multiple accounts in a rails app, or a ruby app that uses concurrency in some way, then you need to be using the thread safe configuration.
The vicious bug
This was one of the most difficult bugs to diagnose that I can remember. I work on a large rails application that collates data from various social networks, for multiple accounts. Each application account can have one or more Twitter accounts linked to it, and data is retrieved from the Twitter streams multiple times a day.
One day, we noticed that the Twitter data was crossing over between accounts in our production environment. The tweets that should have been in one account were appearing for a totally separate account. What made it worse was that it seemed to be random; the bug only appeared on some occasions, and it worked perfectly on others.
I tried replicating the issue on my development PC, but couldn't get it to occur. As much as I tried to make it fail, the API communication appeared to be working perfectly. I spent hours walking through the code, line-by-line, writing tests to try and reproduce the problem. No luck.
I then took a look at the configuration of sferik's Twitter gem, which we use for communicating with the twitter API:
Twitter.configure do |config| config.consumer_key = YOUR_CONSUMER_KEY config.consumer_secret = YOUR_CONSUMER_SECRET config.oauth_token = YOUR_OAUTH_TOKEN config.oauth_token_secret = YOUR_OAUTH_TOKEN_SECRET end
While looking at this code, it suddenly hit me: the configuration is a class method called on a globally available module. That essentially means that a global variable is hiding behind the scenes.
It then occurred to me what was happening on the application level. The configuration details for one Twitter account were being passed in, and then at some point during the API communication, another thread or process was reconfiguring the global
The thread safe way
It turned out that the standard way of using the Twitter gem is not thread safe. This is actually addressed in the README, but you have to scroll quite a long way down. Most people would grab the configuration from the Quick Start section, which is probably fine for most people, and is exactly what we did. The thread safe way of configuring a Twitter client is:
client = Twitter::Client.new( consumer_key: "an application's consumer key", consumer_secret: "an application's consumer secret", oauth_token: "a user's access token", oauth_token_secret: "a user's access secret" )
You then keep a hold of the
client and make the relevant calls on that rather than the globally available
The problem was one relating to evolution. We didn't start with Sidekiq and concurrent Twitter API retrieval, this grew gradually over time. I thought I'd share it just in case it helps someone else who may be banging their head against a wall at this very moment.