It's an HTTParty and Everyone Is Invited!

So I’ve made a boatload of gems that consume web services (twitter, lastfm, magnolia, delicious, google reader, google analytics). Each time I get a bit better at it and learn something new. I pretty much am only interested in consuming restful api’s and over time I’ve started to see a pattern. The other day I thought, wouldn’t it be nice if that pattern were wrapped up as a present for me (and others) to use? The answer is yes and it is named HTTParty.

The Pattern

Every web service related gem I’ve written makes requests and parses responses into ruby objects. So first let’s start with requests. The request methods that you make the most use of are get and post, with put and delete occasionally sliding in. I don’t know about you but I constantly forget how to use net/http. First, make the http, then the request. Is it get or post? How do I get the response body? Maybe I’m forgetful, but I always needed to have net/http’s rdoc open when working with it. Not anymore, though, cause now I just HTTParty like it’s 1999.

A Princely Example

To find something that I haven’t written a gem for, I hopped over to programmable web’s API directory. I chose Who is my representative‘s API because, frankly, I didn’t know it existed and it was really basic. So let’s say we want to get who my rep is:

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
end

puts Representative.get('http://whoismyrepresentative.com/whoismyrep.php?zip=46544').inspect
# "<result n='1' ><rep name='Joe Donnelly' state='IN' district='2' phone='(202) 225-3915' office='1218 Longworth' link='http://donnelly.house.gov/' /></result>"

Yep, that is it. Include HTTParty and you are good to go. So, that was easy. Now we can make requests, but our response was just plain old xml. We want ruby objects! Let’s require rexml or install hpricot or libxml-ruby next, right? Wrong! Just set the format.

Automatically Parse XML

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  format :xml
end

puts Representative.get('http://whoismyrepresentative.com/whoismyrep.php?zip=46544').inspect
# {"result"=>{"n"=>"1", "rep"=>{"name"=>"Joe Donnelly", "district"=>"2", "office"=>"1218 Longworth", "phone"=>"(202) 225-3915", "link"=>"http://donnelly.house.gov/", "state"=>"IN"}}}

Yep, I’m not joking. That works, but let’s wrap things up a little bit and make a prettier API for the developer that will eventually use our Representative library.

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  format :xml
  
  def self.find_by_zip(zip)
    get('http://whoismyrepresentative.com/whoismyrep.php', :query => {:zip => zip})
  end
end

puts Representative.find_by_zip(46544).inspect
# {"result"=>{"n"=>"1", "rep"=>{"name"=>"Joe Donnelly", "district"=>"2", "office"=>"1218 Longworth", "phone"=>"(202) 225-3915", "link"=>"http://donnelly.house.gov/", "state"=>"IN"}}}

There, that is a little better. One simple module include (HTTParty) and we can make requests and automatically get our xml responses parsed. Not to mention it’s so easy my mom could do it. What? Oh, you want to see JSON. Sure, no problem.

Automatically Parse JSON

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  format :json
  
  def self.find_by_zip(zip)
    get('http://whoismyrepresentative.com/whoismyrep.php', :query => {:zip => zip, :output => 'json'})
  end
end

puts Representative.find_by_zip(46544).inspect
# {"results"=>[{"name"=>"Joe Donnelly", "district"=>"2", "office"=>"1218 Longworth", "phone"=>"(202) 225-3915", "link"=>"http://donnelly.house.gov/", "state"=>"IN"}]}

Holla! You thought you had me but you didn’t. Let’s make our example a little bit more complicated and add another method to get all the reps by name.

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  format :json
  
  def self.find_by_zip(zip)
    get('http://whoismyrepresentative.com/whoismyrep.php', :query => {:zip => zip, :output => 'json'})
  end
  
  def self.get_all_by_name(last_name)
    get('http://whoismyrepresentative.com/getall_reps_byname.php', :query => {:lastname => last_name, :output => 'json'})
  end
end

puts Representative.get_all_by_name('Donnelly').inspect
# {"results"=>[{"district"=>"2", "last"=>"Donnelly", "first"=>"Joe", "state"=>"IN", "party"=>"D"}]}

Notice any problems with that? I do. I’m repeating the domain and the output format in each request. Let’s fix that.

Helpers To DRY Things Up

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  base_uri 'whoismyrepresentative.com'
  default_params :output => 'json'
  format :json
  
  def self.find_by_zip(zip)
    get('/whoismyrep.php', :query => {:zip => zip})
  end
  
  def self.get_all_by_name(last_name)
    get('/getall_reps_byname.php', :query => {:lastname => last_name})
  end
end

puts Representative.get_all_by_name('Donnelly').inspect
# {"results"=>[{"district"=>"2", "last"=>"Donnelly", "first"=>"Joe", "state"=>"IN", "party"=>"D"}]}

I used base_uri to remove the duplication of the domain and default_params to automatically append :output => ‘json’ to each request. The previous examples give you a really good example of what HTTParty can do, but there is one last example I’ll show.

HTTP Authentication

The only thing we haven’t covered is authentication. API keys are simple, just add them to default_params I showed in the last example, but what about http authentication? Twitter uses http authentication, so our next example will use them.

require 'rubygems'
require 'httparty'

class Twitter
  include HTTParty
  base_uri 'twitter.com'
  basic_auth 'username', 'password'
end

puts Twitter.post('/statuses/update.json', :query => {:status => "It's an HTTParty and everyone is invited!"}).inspect
# {"user"=>{"name"=>"Snitch Test", "url"=>nil, "id"=>808074, "description"=>nil, "protected"=>true, "screen_name"=>"snitch_test", "followers_count"=>1, "location"=>"Hollywood, CA", "profile_image_url"=>"http://static.twitter.com/images/default_profile_normal.png"}, "favorited"=>nil, "truncated"=>false, "text"=>"It's an HTTParty and everyone is invited!", "id"=>870885871, "in_reply_to_user_id"=>nil, "in_reply_to_status_id"=>nil, "source"=>"web", "created_at"=>"Mon Jul 28 20:07:52 +0000 2008"}

Cool. Wait, HTTP Authentication has to go in the class? No silly, Trix are for kids! The basic_auth method is just a class method so you can use it wherever class methods are acceptable. Try this on for size:

require 'rubygems'
require 'httparty'

class Twitter
  include HTTParty
  base_uri 'twitter.com'

  def initialize(user, pass)
    self.class.basic_auth user, pass
  end

  def post(text)
    self.class.post('/statuses/update.json', :query => {:status => text})
  end
end

puts Twitter.new('username', 'password').post("It's an HTTParty and everyone is invited!").inspect
# {"user"=>{"name"=>"Snitch Test", "url"=>nil, "id"=>808074, "description"=>nil, "protected"=>true, "screen_name"=>"snitch_test", "followers_count"=>1, "location"=>"Hollywood, CA", "profile_image_url"=>"http://static.twitter.com/images/default_profile_normal.png"}, "favorited"=>nil, "truncated"=>false, "text"=>"It's an HTTParty and everyone is invited!", "id"=>870885871, "in_reply_to_user_id"=>nil, "in_reply_to_status_id"=>nil, "source"=>"web", "created_at"=>"Mon Jul 28 20:07:52 +0000 2008"}

Miscellaneous

Conclusion

Ok, so that was a really long introduction, but hopefully it was helpful. I’ve also included examples in the gem for those who want to venture more (twitter, delicious, amazon associates web services, most basic usage). Also, don’t be afraid of the code as it doesn’t have much (< 140 lines at the moment).

39 Comments

  1. This is very hot, John! I’m excited to see a simple, usable DSL for creating REST clients and to see what people do with it now that it’s here.

  2. Uh, why’s basic_auth a class method? What happens if I want two instances of the Twitter class with different credentials?

  3. This gem is great John! I think I’ll rewrite my Piwik API client gem to use HTTParty instead of rest-client, your solution seems a lot cleaner :) At least it’s still in a very alpha stage, so not a lot of rewriting to do.

    Congrats again for HTTParty!

  4. @Rein – Thanks!

    @Tom, I see what your saying, but I would do everything with the first one and then move on to the second set of credentials. I can’t think of a better way as all the rest of it is class methods. Any suggestions?

    @Rodrigo – Thanks! Let me know if you run into rode blocks with HTTParty.

  5. @Tom – Got an idea. I’ll make an optional auth option that can go along with any request. If that is present in the options, it will use that instead of the class level auth.

  6. This rocks! Thanks.

  7. @John — This is awesome!

  8. I’ve been working on a similar project, Resourceful. It’s essentially the same thing: An http library to make ReST clients easier to write. It has a few additional features, like caching and redirect handling. Its interface is written in such a way that its easy to write more libraries that use it. I’m using it in production in a couple places, and am currently working on a datamapper ReST adapter that utilizes it.

    http://resourceful.rubyforge.org/

  9. Great. This is the kind of dead simple useful code I love.

    Suggestion: infer media types based on header or extension conventions.

  10. Hey thanks for this awesome tool!

    makes it 10.times easier to work with APIs

  11. @Paul – Cool, I’ll check it out.

    @Jason – Good idea. I’ll look into that.

  12. Cool shit dude, much nicer than RestClient. Though I’m not sure your basic auth example with object instances is threadsafe.

  13. @rick – Yeah I’m pretty sure it’s not too though I haven’t messed with threads much. I’m thinking I’ll add an :auth option to go along with :query, :body and :headers when making requests.

  14. Hey, cool gem. Please add support for logging capabilities. I like to be able to see in development mode where I am getting / posting and what parameters I am passing.

  15. Also, it could be my fault but it doesn’t seem like a post actually sends parameters properly. Browsing the code it appears that you simply tack on query parameters onto the end of the uri. This isn’t really how POST parameters should be sent.

  16. Very nice!

  17. Great gem!!

    But how do you use it through a proxy?

  18. Looks good. Have you thought about including HTTPS into this?

  19. @Nathan – Logging is a good idea. Posts use the :body option, not :query which just appends stuff to the query string. :body should be just a string.

    @Chris – I’ve never had to use anything through a proxy. If you can give me an example of how you do it and how I can test it using a proxy, I could maybe figure it out and add it in.

    @Steve – HTTPS is in. If the uri has port 443 it automatically turns https on.

  20. John, re Tom’s comment about auth, I think the right thing to do is not try and shoehorn it all into class methods. If people want a persistent definition that sticks around, there’s nothing stopping them from doing something like this, which I think I’d find a better api:

    Twitter = Httparty.new do
    base_uri ‘twitter.com’
    basic_auth ‘username’, ‘password’
    end

  21. @John – I often just want to POST parameters within the body of the request. I realize I could do { :foo => ‘bar’ }.to_query but is it possible you can make it so that if the body option is a hash that it will convert the hash to query string?

    i.e

    Example.post ‘/url’, :body => { :foo => ‘bar’ }

    adds ‘foo=bar’ to the post body.

  22. @Nathan – You ask too much. I kid, I kid. Yeah, that was pure laziness on my part. Good idea.

  23. The auth issue is fixed. There is now a :basic_auth key that can be passed to any request.

    Also, the :query and :body keys both take a query string or a hash now. Previously, :query only accepted a hash and :body only string.

    I released v0.1.1.

  24. @John – Thanks John, I appreciate the body accepting a hash fix. I can now integrate this into my project perfectly.

  25. This is great, can’t wait to use it the next time I need to consume a REST API!

    I was initially concerned about basic_auth being a class method, as I might need multiple instances for different users, perhaps even in a multi-threaded application. But your fix sounds like it should solve this problem. The only downside I see is that the :basic_auth key now needs to be passed into each individual request, making it less DRY. Being able to call “get” or “post” on an instance rather than the class would allow us to specify the basic_auth settings once in the initialize method, removing duplication. Definitely not the end of the world — either way, this is a lot simpler than using Net::HTTP or similar… :)

  26. This is a very nice abstraction of all your API consumer gems. Very cool.

  27. @Dr. Nic: Thanks! It’s all made possible by newgem. :)

  28. Wow, great gem! I just used to whip up a [Ruby wrapper for Google Translate](http://gist.github.com/3866/) in no less than 5 minutes!

  29. @Bryce – Nice! Glad you’re a fan.

  30. And if you are lazy and like ActiveRecord accessors like me you can do

    
    class Hash
      def method_missing(key)
        self[key.to_s] || super
      end
    end
    

    Then you can just do

    
    &gt;&gt; reps = Representative.find_by_zip(60657)
    ...
    &gt;&gt; reps.results.first.name
    # =&gt; "Rahm Emanuel"
    
  31. this is great, John. Thanks for your work. One thing that might be helpful is putting a rescue in parse_response in case, for example, twitter is down and an error is returned. Something like:
    rescue raise StandardError, body end

    Your gem has saved me lots of time.

  32. @Underpants – Nice.

    @Steve Odom – Yeah, I have a ticket open for handling response codes.

  33. The DSL API looks very nice.

    However, is a module the right encapsulation? From all the examples given, it seems like a base class would be more fitting.

    I think the question to ask is “will I use this to augment classes I already have, or will I use it to create new classes?” If usually the later than a base class works best — It also makes the design easier, btw, no self.included() tricks.

  34. I really enjoy how you’ve implemented HTTParty as a decorated mix-in and not as a hierarchical parent-class. Great work!

  35. This is totally my new favorite gem. Awesome, John.

  36. Raffaele Tesi Raffaele Tesi

    Aug 07, 2008

    @Chris
    simple hack to handle proxy


    httparty.rb
    private
    ###def http(uri)
    def http(uri,p_host=nil,p_port=nil)
    if @http.blank?

    1. @http = Net::HTTP.new(uri.host, uri.port)
      @http = Net::HTTP.new(uri.host, uri.port,p_host,p_port)
      -—————————————————————-
    1. httparty.rb, method send_request
      #response= http(uri).request(request)
      response=http(uri,options[:p_address],options[:p_port]).request(request)
      ####################

    Now just add “:p_address=>’yourproxy’,:p_port=>’yourport’” to the options hash when needed, eg.


    Representative.get(‘http://whoismyrepresentative.com/whoismyrep.php?zip=46544’,:p_address=>’yourproxy’, :p_port=>’yourport’).inspect

  37. Franxxk Franxxk

    Aug 10, 2008

    About proxy…
    If thing a class method is better… of course method http must change also

    Class MyRest
    include HTTParty
    http_proxy ‘myProxy’, 1080

    I’ve also implemented RESTFul object access, and i observe for google health for example, that redirect (302) happened. then i write my get/post method than follow redirect for a depth of 2 by default.
    Lots of us would apreciate such feature for HTTParty…

  38. There appears to be to be a bug if the XML returned has a tag with no text value and just other tags.

    For example

    5 whatever

    patterns is returned in the hash as nil so it doesn’t traverse it…

  39. Looks like a fix was added to a fork of your master but not yet merged.

    http://github.com/wesmaldonado/httparty/commits/master

    http://github.com/wesmaldonado/httparty/commit/28395ae393f21eeef50d698e373105ce63e54a1e

Sorry, comments are closed for this article to ease the burden of pruning spam.

About

Authored by John Nunemaker (Noo-neh-maker), a web developer and programmer who has fallen deeply in love with Ruby. More about John.

Syndication

Feed IconRailsTips Articles - An assortment of howto's and thoughts on Ruby and Rails.

Feed IconRails Quick Tips - Ruby and Rails related links.