July 29, 2008

Posted by John

Tagged gems, httparty, rest, twitter, and web services

Older: Raking /etc/hosts For Sweeter Subdomainage

Newer: Google App Engine Hack Day

It's an HTTParty and Everyone Is Invited!

So I’ve made a boatload of gems that consume web services (twitter, lastfm, magnolia, delicious, google reader, google analytics). Each time I get a bit better at it and learn something new. I pretty much am only interested in consuming restful api’s and over time I’ve started to see a pattern. The other day I thought, wouldn’t it be nice if that pattern were wrapped up as a present for me (and others) to use? The answer is yes and it is named HTTParty.

The Pattern

Every web service related gem I’ve written makes requests and parses responses into ruby objects. So first let’s start with requests. The request methods that you make the most use of are get and post, with put and delete occasionally sliding in. I don’t know about you but I constantly forget how to use net/http. First, make the http, then the request. Is it get or post? How do I get the response body? Maybe I’m forgetful, but I always needed to have net/http’s rdoc open when working with it. Not anymore, though, cause now I just HTTParty like it’s 1999.

A Princely Example

To find something that I haven’t written a gem for, I hopped over to programmable web’s API directory. I chose Who is my representative‘s API because, frankly, I didn’t know it existed and it was really basic. So let’s say we want to get who my rep is:

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
end

puts Representative.get('http://whoismyrepresentative.com/whoismyrep.php?zip=46544').inspect
# "<result n='1' ><rep name='Joe Donnelly' state='IN' district='2' phone='(202) 225-3915' office='1218 Longworth' link='http://donnelly.house.gov/' /></result>"

Yep, that is it. Include HTTParty and you are good to go. So, that was easy. Now we can make requests, but our response was just plain old xml. We want ruby objects! Let’s require rexml or install hpricot or libxml-ruby next, right? Wrong! Just set the format.

Automatically Parse XML

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  format :xml
end

puts Representative.get('http://whoismyrepresentative.com/whoismyrep.php?zip=46544').inspect
# {"result"=>{"n"=>"1", "rep"=>{"name"=>"Joe Donnelly", "district"=>"2", "office"=>"1218 Longworth", "phone"=>"(202) 225-3915", "link"=>"http://donnelly.house.gov/", "state"=>"IN"}}}

Yep, I’m not joking. That works, but let’s wrap things up a little bit and make a prettier API for the developer that will eventually use our Representative library.

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  format :xml
  
  def self.find_by_zip(zip)
    get('http://whoismyrepresentative.com/whoismyrep.php', :query => {:zip => zip})
  end
end

puts Representative.find_by_zip(46544).inspect
# {"result"=>{"n"=>"1", "rep"=>{"name"=>"Joe Donnelly", "district"=>"2", "office"=>"1218 Longworth", "phone"=>"(202) 225-3915", "link"=>"http://donnelly.house.gov/", "state"=>"IN"}}}

There, that is a little better. One simple module include (HTTParty) and we can make requests and automatically get our xml responses parsed. Not to mention it’s so easy my mom could do it. What? Oh, you want to see JSON. Sure, no problem.

Automatically Parse JSON

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  format :json
  
  def self.find_by_zip(zip)
    get('http://whoismyrepresentative.com/whoismyrep.php', :query => {:zip => zip, :output => 'json'})
  end
end

puts Representative.find_by_zip(46544).inspect
# {"results"=>[{"name"=>"Joe Donnelly", "district"=>"2", "office"=>"1218 Longworth", "phone"=>"(202) 225-3915", "link"=>"http://donnelly.house.gov/", "state"=>"IN"}]}

Holla! You thought you had me but you didn’t. Let’s make our example a little bit more complicated and add another method to get all the reps by name.

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  format :json
  
  def self.find_by_zip(zip)
    get('http://whoismyrepresentative.com/whoismyrep.php', :query => {:zip => zip, :output => 'json'})
  end
  
  def self.get_all_by_name(last_name)
    get('http://whoismyrepresentative.com/getall_reps_byname.php', :query => {:lastname => last_name, :output => 'json'})
  end
end

puts Representative.get_all_by_name('Donnelly').inspect
# {"results"=>[{"district"=>"2", "last"=>"Donnelly", "first"=>"Joe", "state"=>"IN", "party"=>"D"}]}

Notice any problems with that? I do. I’m repeating the domain and the output format in each request. Let’s fix that.

Helpers To DRY Things Up

require 'rubygems'
require 'httparty'

class Representative
  include HTTParty
  base_uri 'whoismyrepresentative.com'
  default_params :output => 'json'
  format :json
  
  def self.find_by_zip(zip)
    get('/whoismyrep.php', :query => {:zip => zip})
  end
  
  def self.get_all_by_name(last_name)
    get('/getall_reps_byname.php', :query => {:lastname => last_name})
  end
end

puts Representative.get_all_by_name('Donnelly').inspect
# {"results"=>[{"district"=>"2", "last"=>"Donnelly", "first"=>"Joe", "state"=>"IN", "party"=>"D"}]}

I used base_uri to remove the duplication of the domain and default_params to automatically append :output => ‘json’ to each request. The previous examples give you a really good example of what HTTParty can do, but there is one last example I’ll show.

HTTP Authentication

The only thing we haven’t covered is authentication. API keys are simple, just add them to default_params I showed in the last example, but what about http authentication? Twitter uses http authentication, so our next example will use them.

require 'rubygems'
require 'httparty'

class Twitter
  include HTTParty
  base_uri 'twitter.com'
  basic_auth 'username', 'password'
end

puts Twitter.post('/statuses/update.json', :query => {:status => "It's an HTTParty and everyone is invited!"}).inspect
# {"user"=>{"name"=>"Snitch Test", "url"=>nil, "id"=>808074, "description"=>nil, "protected"=>true, "screen_name"=>"snitch_test", "followers_count"=>1, "location"=>"Hollywood, CA", "profile_image_url"=>"http://static.twitter.com/images/default_profile_normal.png"}, "favorited"=>nil, "truncated"=>false, "text"=>"It's an HTTParty and everyone is invited!", "id"=>870885871, "in_reply_to_user_id"=>nil, "in_reply_to_status_id"=>nil, "source"=>"web", "created_at"=>"Mon Jul 28 20:07:52 +0000 2008"}

Cool. Wait, HTTP Authentication has to go in the class? No silly, Trix are for kids! The basic_auth method is just a class method so you can use it wherever class methods are acceptable. Try this on for size:

require 'rubygems'
require 'httparty'

class Twitter
  include HTTParty
  base_uri 'twitter.com'

  def initialize(user, pass)
    self.class.basic_auth user, pass
  end

  def post(text)
    self.class.post('/statuses/update.json', :query => {:status => text})
  end
end

puts Twitter.new('username', 'password').post("It's an HTTParty and everyone is invited!").inspect
# {"user"=>{"name"=>"Snitch Test", "url"=>nil, "id"=>808074, "description"=>nil, "protected"=>true, "screen_name"=>"snitch_test", "followers_count"=>1, "location"=>"Hollywood, CA", "profile_image_url"=>"http://static.twitter.com/images/default_profile_normal.png"}, "favorited"=>nil, "truncated"=>false, "text"=>"It's an HTTParty and everyone is invited!", "id"=>870885871, "in_reply_to_user_id"=>nil, "in_reply_to_status_id"=>nil, "source"=>"web", "created_at"=>"Mon Jul 28 20:07:52 +0000 2008"}

Miscellaneous

Install: sudo gem install httparty
Rubyforge: http://httparty.rubyforge.org
Github: http://github.com/jnunemaker/httparty
Lighthouse: http://jnunemaker.lighthouseapp.com/projects/14842-httparty/overview

Conclusion

Ok, so that was a really long introduction, but hopefully it was helpful. I’ve also included examples in the gem for those who want to venture more (twitter, delicious, amazon associates web services, most basic usage). Also, don’t be afraid of the code as it doesn’t have much (< 140 lines at the moment).

39 Comments

Rein Henrichs
Jul 29, 2008

This is very hot, John! I’m excited to see a simple, usable DSL for creating REST clients and to see what people do with it now that it’s here.
Tom
Jul 29, 2008

Uh, why’s basic_auth a class method? What happens if I want two instances of the Twitter class with different credentials?
Rodrigo
Jul 29, 2008

This gem is great John! I think I’ll rewrite my Piwik API client gem to use HTTParty instead of rest-client, your solution seems a lot cleaner :) At least it’s still in a very alpha stage, so not a lot of rewriting to do.

Congrats again for HTTParty!
John Nunemaker
Jul 29, 2008

@Rein – Thanks!

@Tom, I see what your saying, but I would do everything with the first one and then move on to the second set of credentials. I can’t think of a better way as all the rest of it is class methods. Any suggestions?

@Rodrigo – Thanks! Let me know if you run into rode blocks with HTTParty.
John Nunemaker
Jul 29, 2008

@Tom – Got an idea. I’ll make an optional auth option that can go along with any request. If that is present in the options, it will use that instead of the class level auth.
John Topley
Jul 29, 2008

This rocks! Thanks.
Michael Guterl
Jul 29, 2008

@John — This is awesome!
Paul
Jul 29, 2008

I’ve been working on a similar project, Resourceful. It’s essentially the same thing: An http library to make ReST clients easier to write. It has a few additional features, like caching and redirect handling. Its interface is written in such a way that its easy to write more libraries that use it. I’m using it in production in a couple places, and am currently working on a datamapper ReST adapter that utilizes it.

http://resourceful.rubyforge.org/
Jason Watkins
Jul 29, 2008

Great. This is the kind of dead simple useful code I love.

Suggestion: infer media types based on header or extension conventions.
Joseph Hsu
Jul 29, 2008

Hey thanks for this awesome tool!

makes it 10.times easier to work with APIs
John Nunemaker
Jul 29, 2008

@Paul – Cool, I’ll check it out.

@Jason – Good idea. I’ll look into that.
rick
Jul 29, 2008

Cool shit dude, much nicer than RestClient. Though I’m not sure your basic auth example with object instances is threadsafe.
John Nunemaker
Jul 30, 2008

@rick – Yeah I’m pretty sure it’s not too though I haven’t messed with threads much. I’m thinking I’ll add an :auth option to go along with :query, :body and :headers when making requests.
Nathan
Jul 30, 2008

Hey, cool gem. Please add support for logging capabilities. I like to be able to see in development mode where I am getting / posting and what parameters I am passing.
Nathan
Jul 30, 2008

Also, it could be my fault but it doesn’t seem like a post actually sends parameters properly. Browsing the code it appears that you simply tack on query parameters onto the end of the uri. This isn’t really how POST parameters should be sent.
Travis
Jul 30, 2008

Very nice!
Chris
Jul 30, 2008

Great gem!!

But how do you use it through a proxy?
Steve
Jul 30, 2008

Looks good. Have you thought about including HTTPS into this?
John Nunemaker
Jul 30, 2008

@Nathan – Logging is a good idea. Posts use the :body option, not :query which just appends stuff to the query string. :body should be just a string.

@Chris – I’ve never had to use anything through a proxy. If you can give me an example of how you do it and how I can test it using a proxy, I could maybe figure it out and add it in.

@Steve – HTTPS is in. If the uri has port 443 it automatically turns https on.
Jamie
Jul 30, 2008

John, re Tom’s comment about auth, I think the right thing to do is not try and shoehorn it all into class methods. If people want a persistent definition that sticks around, there’s nothing stopping them from doing something like this, which I think I’d find a better api:

Twitter = Httparty.new do
base_uri ‘twitter.com’
basic_auth ‘username’, ‘password’
end
Nathan
Jul 30, 2008

@John – I often just want to POST parameters within the body of the request. I realize I could do { :foo => ‘bar’ }.to_query but is it possible you can make it so that if the body option is a hash that it will convert the hash to query string?

i.e

Example.post ‘/url’, :body => { :foo => ‘bar’ }

adds ‘foo=bar’ to the post body.
John Nunemaker
Jul 30, 2008

@Nathan – You ask too much. I kid, I kid. Yeah, that was pure laziness on my part. Good idea.
John Nunemaker
Jul 31, 2008

The auth issue is fixed. There is now a :basic_auth key that can be passed to any request.

Also, the :query and :body keys both take a query string or a hash now. Previously, :query only accepted a hash and :body only string.

I released v0.1.1.
Nathan
Jul 31, 2008

@John – Thanks John, I appreciate the body accepting a hash fix. I can now integrate this into my project perfectly.
Mirko Froehlich
Aug 01, 2008

This is great, can’t wait to use it the next time I need to consume a REST API!

I was initially concerned about basic_auth being a class method, as I might need multiple instances for different users, perhaps even in a multi-threaded application. But your fix sounds like it should solve this problem. The only downside I see is that the :basic_auth key now needs to be passed into each individual request, making it less DRY. Being able to call “get” or “post” on an instance rather than the class would allow us to specify the basic_auth settings once in the initialize method, removing duplication. Definitely not the end of the world — either way, this is a lot simpler than using Net::HTTP or similar… :)
Dr Nic
Aug 03, 2008

This is a very nice abstraction of all your API consumer gems. Very cool.
John Nunemaker
Aug 03, 2008

@Dr. Nic: Thanks! It’s all made possible by newgem. :)
Bryce
Aug 04, 2008

Wow, great gem! I just used to whip up a [Ruby wrapper for Google Translate](http://gist.github.com/3866/) in no less than 5 minutes!
John Nunemaker
Aug 04, 2008

@Bryce – Nice! Glad you’re a fan.

UnderpantsGnome

Aug 04, 2008

And if you are lazy and like ActiveRecord accessors like me you can do


class Hash
  def method_missing(key)
    self[key.to_s] || super
  end
end

Then you can just do


&gt;&gt; reps = Representative.find_by_zip(60657)
...
&gt;&gt; reps.results.first.name
# =&gt; "Rahm Emanuel"

steve odom
Aug 04, 2008

this is great, John. Thanks for your work. One thing that might be helpful is putting a rescue in parse_response in case, for example, twitter is down and an error is returned. Something like:
rescue raise StandardError, body end

Your gem has saved me lots of time.
John Nunemaker
Aug 04, 2008

@Underpants – Nice.

@Steve Odom – Yeah, I have a ticket open for handling response codes.
7rans
Aug 05, 2008

The DSL API looks very nice.

However, is a module the right encapsulation? From all the examples given, it seems like a base class would be more fitting.

I think the question to ask is “will I use this to augment classes I already have, or will I use it to create new classes?” If usually the later than a base class works best — It also makes the design easier, btw, no self.included() tricks.
J. Ryan Sobol
Aug 05, 2008

I really enjoy how you’ve implemented HTTParty as a decorated mix-in and not as a hierarchical parent-class. Great work!
Brian Dainton
Aug 06, 2008

This is totally my new favorite gem. Awesome, John.
Raffaele Tesi
Aug 07, 2008
@Chris
simple hack to handle proxy

httparty.rb
private
###def http(uri)
def http(uri,p_host=nil,p_port=nil)
if @http.blank?
1. @http = Net::HTTP.new(uri.host, uri.port)
  @http = Net::HTTP.new(uri.host, uri.port,p_host,p_port)
  -—————————————————————-
1. httparty.rb, method send_request
  #response= http(uri).request(request)
  response=http(uri,options[:p_address],options[:p_port]).request(request)
  ####################
Now just add “:p_address=>’yourproxy’,:p_port=>’yourport’” to the options hash when needed, eg.

Representative.get(‘http://whoismyrepresentative.com/whoismyrep.php?zip=46544’,:p_address=>’yourproxy’, :p_port=>’yourport’).inspect
Franxxk
Aug 10, 2008

About proxy…
If thing a class method is better… of course method http must change also

Class MyRest
include HTTParty
http_proxy ‘myProxy’, 1080
…
…

I’ve also implemented RESTFul object access, and i observe for google health for example, that redirect (302) happened. then i write my get/post method than follow redirect for a depth of 2 by default.
Lots of us would apreciate such feature for HTTParty…
Henry H
Aug 12, 2008

There appears to be to be a bug if the XML returned has a tag with no text value and just other tags.

For example
5 whatever
patterns is returned in the hash as nil so it doesn’t traverse it…
Henry H
Aug 12, 2008

Looks like a fix was added to a fork of your master but not yet merged.

http://github.com/wesmaldonado/httparty/commits/master

http://github.com/wesmaldonado/httparty/commit/28395ae393f21eeef50d698e373105ce63e54a1e