When should I start a REST initiative

Restfulie’s release, centered on hypermedia support, got a lot of attention back to not letting go the HATEOAS idea and the old question arrives again: is it worthy to invest money or time building a fully REST system in my company?

A full REST architecture imply in many choices that some prefer to leave out, but it is interesting to see how people reacted to REST in the last few years.

Using google insights, the first thing to note – a biased query due to the selected keywords? – a continuing increase of interest in REST since 2004 in the programming area.

Surely, this can not be accepted as a single ’soa is dead, long live rest’. In this result for the query, red means ’soa’ and blue means ‘rest’.

A second search including ‘web services’ and ‘web service’ shows a decline for such words. For those who do not consider REST as a web service due to the general notion of WS being related to SOAP and WS-stacks, this is a positive aspect. The following result contains green and yellow meaning ‘webservices’, blue for ‘rest’ and red for ’soa’.

if you compare searches for ejb, soap, corba and rest (blue is rest):

Finally, comparing those technologies to the growth of programming searches, rest is the only one whose growth is bigger than programming searches average:

If you are looking for a contemporaneous architectural style which is growing in its adoption, google seems to point you to REST. This information gives stronger hope to those who are putting their time, money or energy into REST architectures: they might have picked a good path.

A technology evolves faster as more people start using it. Although there is a long way to go with REST and its hypermedia features, it’s the only line going up.

When friends and clients ask if its time to try and learn it… definately.

note: this post is not about rest being good, or better than any other solution compared, just a collection of interesting outcomes from developer’s searches. Remember: there is no silver bullet.

Transactions do not exist in a Restful world…

Due to the last posts on infoq related to Restfulie, my work at Caelum Objects involved a presentation at one client, “Beginning a REST initiative” (based on Ian’s work) and the question came up: “but how do I control transactions without a custom software stack to help me?”

The answer was, “you do not need to”.

Restwiki has an old entry on how to implement transaction support through http using some non-standard http headers.

The idea was not new, as Roy Fielding mentioned on an old mail that this extra http header could be a solution and later seemed to change his mind about it, according to an infoq news.

In practice most ideas are based on a transaction being a resource named “Transaction”: an idea heavily based on HTTP and URIs, but forgetting about HATEOAS – again.

In the human web, how does one buys some products? Every product is added to the shopping basket, which then generates the order. Does the user creates a transaction before processing his order?

The human being behing the computer did not create a transaction: the browser is even unaware of that concept, but hyperlinks given by the server guided the client through this “transaction”. In this case, where the typical “REST” solution would create a “Transaction” resource and use the non-standard header to support it, a Restfulie one creates a shopping basket:

	Typical “REST” approach	Restfulie
sequence	POST /transaction POST /product * POST /product *	POST /basket POST /basket/:id/product POST /basket/:id/product
commit	POST /transaction/commit *	PUT /basket/:id/payment
rollback	DELETE /transaction/ *	DELETE /basket/:id

* with non-standard http header

The standard way of thinking about transactions is to not use HATEOAS and believe that transactions are resources by themselves. Transactions are not resources, but a tool to implement ACID in your (i.e.) databases, not in a web system.

In our example, an order creation maps to internal transactions. In a bank example, a Transfer resource would map to the internal transaction.

By renaming the “transaction” to the real objective of that transaction, one can better map meaningful URI’s to resources.

Note that these are only the advantages of valuing the use of URIs over non-standard http headers (manifest hint?): there is no loss of visibility to layers between the client and the server.

But now one might argue that there are too many entry points. Actually, both implementations contain the same number of “entry” points if there is no hypermedia support: 4. Too many entry points should not be called “entry” points. (entry-hell pattern?)

But do we, in the human web, type in URIs as we go further with our online “transaction”? Do we type in URIs as we do a two-step flight and hotel booking process?

If the entry point POST /basket answers with a:


Header
Location: http://caelumobjects.com/basket/5
Content
<basket>
<link rel="products" href="http://caelumobjects.com/basket/5/products" />
<link rel="coupon" href="http://caelumobjects.com/basket/5/coupon" />
<link rel="pay" href="http://caelumobjects.com/basket/5/payment" />
<link rel="cancel" href="http://caelumobjects.com/basket/5" />
</basket>


Note that our basket – our transaction’s meaning – contains hints on how to operate with it and its relations pretty much in the same way that it would do in the human web: dynamically generated links that allows the server to guide the client throughout the process, eliminating the need to extra “entry-points”.

In a hotel and flight booking system, the booking POST result could be represented as:


Header
Location: http://caelumobjects.com/booking/5
Content
<booking>
<link rel="flights" href="http://caelumobjects.com/booking/5/flights" />
<link rel="hotels" href="http://caelumobjects.com/booking/5/hotels" />
<link rel="pay" href="http://caelumobjects.com/booking/5/payment" />
<link rel="cancel" href="http://caelumobjects.com/booking/5" />
</booking>


Note how the first idea on implementing transactions evolved. From a custom header which interferes with visibility and creates the need for custom built clients and layers to understand this instruction, with no server guidance at all to a system where there is no need to customize your client api or layers and the server guides the user flow through hypermedia, maturing your system.

Transactions should not be called “transactions”. The basket or transfer resource are examples of that: they are typical server side implemented transactions that should be actual resources.

Our basket (and thus transfer) seems to match Roy’s comment at that time:

• “As far as the client is concerned, it is only interacting with one resource at a time even when those interactions overlap asynchronously.”: the basket or the transfer
• “There is no “transaction protocol” aside from whatever agreement mechanism is implemented in the back-end in accordance with the resource semantics (in a separate architecture that we don’t care about here).”: you add products to the list of products form that basket, add some coupons and so on
• “There is no commit protocol other than the presentation of various options to the client at any given point in the application.”: hateoas
• “There is no need for client-side agreement with the transaction protocol because the client is only capable of choosing from the choices provided by the server.”: transaction protocol? no transaction protocol here, just a simple resource handling

Restfulie – as many other rest frameworks -already support the first step (running away from the custom header), but goes further when being “hypermedia centric”, it allows the developer to implement it without any effort.

Being opiniated and forcing the adoption of hypermedia as a way to guide or clients through out our processes might be one step ahead into more web (rest in this case?) friendly world as Ryan Riley pointed out.

HATEOAS, HTTP and URIs allow you to eliminate the concept of transaction management (and web transaction managers) from your systems as we usually think of them. There are two steps to follow:

1. there are no transactions
2. let the server guide you, do not try to guide him with multiple entry points

Hypermedia and dynamic contracts: let my bandwidth rest!

“Break it” to scale!

Many systems contain webpages that are very similar to user “custom pages”, where they can configure what they want to see, and every piece is aggregated from different sources into one single page.

In some cases, those are widget based frameworks as wicket and gwt that can be added to my custom page; in other cases you have aggregating portals.

An example of this kind of application (even though its not configurable) is a retail website containing four sections in its home page: the top 10, my orders, random items, and weird items.

In this case, all information come from the same source, but every part has a different probable validity if it is going to be cached. If the page is served as one big chunck of information, it will always be stale due to the random items section. “My orders” is stale only when I place a new order and, in the same way, the top 10 is only stale if any item is bought and surpasses the number of times the 10th item was bought so far.

One of the main issues with this type of pages which aggregate information from one or many sources with different expire-expectations is that cached versions in proxies and clients become stale faster than it should for some elements: once one of this providing sources publishes new information or is updated, the entire representation becomes stale..

Martin Fowler described once a well spread approach to allow those pages to be partially cached within local proxies and clients, thus sharing requested representations between multiple users.

The approach

Given the coffee scenario, one would create different json representations:

• http://restbucks.com/top_sellers
• http://restbucks.com/my_orders
• http://restbucks.com/weird_items
• http://restbucks.com/random_item

And finally an aggregating page:

<html>
<a class="lazy_load" href="http://restbucks.com/top_sellers">Top sellers</a>
<a class="lazy_load" href="http://restbucks.com/my_orders">My orders</a>
<a class="lazy_load" href="http://restbucks.com/random_items">Random items</a>
<a class="lazy_load" href="http://restbucks.com/weird_items">Weird items</a>

And then, for each lazy_load link, we create a div with its content:

<script>
$('.lazy_load').each(function(link) {
  uri = link.attr('href');
  div = $('


').load(uri); // cache hits!
  link.after(div);
});
</script>
</html>

This allows our proxies to cache each component in our page apart from the page itself: whenever one page’s content becomes stale in a proxy, only part of that page needs update.

In a web were most data can be cached and does not become stale so fast, this technique should usually lessen the amount of data being transfered between client and server.

All one needs to do is properly use the http headers for caching.

Remember that if your client supports either parallel requests to the server and/or keep-alive connection, the results might be even better.

Distributed systems? Linked resources?

Roy Fielding mentions that in the data view in REST systems, “small or medium-grain messages are used for control semantics, but the bulk of application work is accomplished via large-grain messages containing a complete resource representation.”

Pretty much in the same way as with the human web, a distributed system using the web as its infrastructure will gain the same cache benefits as long as they implement correct caching policies through http headers (and correct http verbs).

When your server provides a resource representation linking to a series of other related resources the client and proxies staying on the way will be allowed to cache each and every other resource on its own.

This approach results, again, in changes applied to one resource not affecting cached representations of other resources. An stale representation will not affect those accessing other resources within the same context.

Sometimes the decision whether to change latency for scalability might depend on how you think your clients will use your resources: in the human web mentioned above, the developer knew exactly how its clients would access it.

In distributed systems using REST, guessing how resources will be used can be dangerous as it allows you to tight couple yourself to this behaviour while published resources can and would be used in unforeseen ways.

Roy’s dissertation seems to apply here to balance things: “a protocol that requires multiple interactions per user action, in order to do things like negotiate feature capabilities prior to sending a content response, will be perceptively slower than a protocol that sends whatever is most likely to be optimal first and then provides a list of alternatives for the client to retrieve if the first response is unsatisfactory”.

Giving information that will help most cases is fine and providing links to further resources details allow you to balance between latency and scalability (due to caching) as you wish.

Dynamic contracts

This is only possible because we have signed dynamic contracts with our clients. They expect us to follow some formal format definition (defined in xhtml) and processes. How our processes are presented within our representations is the dynamic part of the contract.

While the fixed part can be validated with the use of schema validators, the dynamic part – the process – which is guided by our server needs to be validated through testing the behaviour of our applications: asserting that hypermedia guided transitions should be reflected in our application state.

Nowadays

On the other hand, many contemporary systems use the POST verb receiving a response including many representations at once or the GET verb without any cache related headers: thus not profiting from the web infrastructure at all. This could changed with one (or both) of the following:

• use the GET verb with cache headers
• use hypermedia and micro formats to describe relations between resources

Using it might present similar results as hypermedia+GET+cache headers in the human web – and some styles might already be providing support for it, although not being a constraint.

Note that in this case hypermedia is not driving the application state, but helping with scalability issues.

Progressive enhancement

Martin notes that this is a kind of progressive enhancement: although its definition is related to accessibility, its control over bandwidth benefits are similar to the approach mentioned ones.

Any other systems that use hyperlinks to “break” representations and scale?

Hypermedia: making it easier to create dynamic contracts

The human web and christmas gifts

You have been buying books at amazon.com for 5 years now: typing www.amazon.com in your browser, searching for your book, adding it to the cart and entering your credit card information.

But this year, on December 15th 2009 something new happens. Amazon has launched an entire new “christmas discount program” and in their front page there is a huge ad notifying their clients about this new item.

How do you react?

“Contract violated! I am not buying anything today.”

The key issue in loosely coupled systems is the ability to evolve one side without implying in any modifications on the other part.

As some Rest guys agree, hypermedia content was the factor which allowed such situations to happen in the human web without clients screaming “i don’t know what to do now that there is a black friday clearance!” or “there is a new link in this page, let me email the ‘webmaster’ and complain about it“.

In the human web, some contracts are agreed upon and validated through end-to-end tests. Some companies will use tools as selenium-rc, webdriver or cucumber to drive their tests and ensure that expected behaviour by their clients does not break with a new release of their software.

Those tests do not validate all content, though, giving space for what is called forward-compatibility: the system is free to create new functionalities without breaking previous expected behaviour.

But my rest-client is not human

In the non-human web, the most well known media type used is xml, although not hypermedia-capable. There are a couple of ways to create forward or backward-compatible schemas that check xml structures, but – unfortunately – usually
fixed schemas will not invest part of its contract in order to making it forward-compatible: its an optional feature.

One option is to create “polymorphic” types through xsd schemas, which will get nasty if your system evolves continuously – not once every year – and you find yourself in a schema-hell situation.

One easy solution is to accept anything in too many places, which seems odd.

What are we missing then? According to Subbu Allamaraju, in RESTful applications, “only a part of the contract can be described statically, and the rest is dynamic and contextual”: you tell your client that they can believe you will not break the statically contract – you might use some schema validation to do that – and it’s up to you on the server side to not do it on the dynamic part.

Some might think it sounds too loose… let’s recall the human web again:

• xhtml allows you to validate your system’s fixed contract
• it’s up to you not to remove an important form used throughout the buying process

So, what are the dynamic parts of my “contract”?

In a RESTful application the contract depends on its context, which is highly affected by three distinct components:

1. your resource’s state

If a person had his application denied to open an account, your resource representation will not offer a “create_loan” request. A denied application is an information regarding its state.

While your company and application evolves, its common to find ourselves in a position where new states appear.

2. your resource’s relations

In a book store (i.e. amazon a few years ago), a book might have a category associated with it so you can access other similar books:

<book>
<name>Rest if you do not want to get tired</name>
<link rel="category" href="http://www.caelumobjects.com/categories/self-help" />
</book>



A couple years later, your system might add extra relations, as "clients which bought this book also recommend"

<book>
 <name>Rest if you do not want to get tired</name>
 <link rel="category" href="http://www.caelumobjects.com/categories/self-help" />
 <link rel="recommendation" href="http://www.caelumobjects.com/books/take-a-shower-with-a-good-soap-if-you-need-to-rest" />
</book>

When your company and application evolves, its common to find ourselves in a position where new relations appear.

3. your resource’s operations

In a REST application, your resource operation’s are represented by HTTP verbs: supporting a new one will not affect clients which use all other available verbs so far.

In the RPC/Webservices world, new operations would be implemented creating new remote procedures or services.

But how can my clients be sure that I will not break the dynamic contract?

Pretty much in the same way that you do in the human web: it’s your word.

In the human web, how do we guarantee that we will not remove or break some functionality the user expects to be there? We end-to-end automatically test its behaviour.
Our word (our tests) is the only reason to rest without worries that we will not break our client’s expectations. The same holds on the non-human web.

The dynamic contract should be throughly tested in order to not break our client’s expectations.

There are other approaches (as client-aware contracts) which might add some extra coupling between both sides.

HTTP+XML+ATOM gives us the possibility to work with both the fixed (schema validated) and dynamic (test validated) contract.

As Bill Burke pointed in a comment, “you can design your XML schemas to be both flexible and backward compatible ” and “companies, users, developers desire this contract”.

That’s the good points of using schemas, but its not everyone that use them in a flexible and backward compatible way. Even those who use might have a little bit of hard time to support it, i.e. having to maintain more than one entry point for each version of their schemas.

That’s when we can use the good points of the schema validation, as Bill pointed out, with the easy evolution advantages of a dynamic contract: as we do in the human web.

By using dynamic contracts as xml+atom following the Must Ignore rules, forward and backward compatibility is gained by default, independent on what the user does – assuming that tests are a must in any solution.

Dynamic contracts also give hints for frameworks, as they guide you on what your user can and can not do or access, but maybe not for tools, in a different fashion of what fixed contracts do: with a fixed schema I would be able to pre-generate my classes, while with dynamic schemas I the framework inject methods.

That’s why we try to take an approach which force programmers to adopt xml+atom. The entry point on the Restfulie framework is loosely evolution.

Its first example, the documentation and its examples do not focus on how easy it is to use nice URIs and the 4 most famous http verbs, but how easy it is to evolve your system using hypermedia and http: uri’s come soon afterwards.

And it seems to be working fine to far, the first developers using it in live systems have already supported hypermedia content as a way to guide clients through their systems.

Restfulie support in dynamic contracts

Matt pulver’s extension to Rails allows one to instantiate types with regards to their active record relations and attributes, but it requires every xml element to be present (strong coupling to the data structure presented by the server).

Using Jeokkarak (korean hashis), Restfulie instantiate objects matching your local data structure, supporting fields defined in your attributes and inserting extra fields for those elements unknown to your model.

For example, if you have a model as:

class Bill
  attr_accessor :value, :to_date
end

And the following xml:

<bill>
  <value>100</value>
  <to-date>10/10/2010</to-date>
  <taxes>0.07</taxes>
</bill>

The result is a dynamic object capable of answering to:

bill = Bill.from_web uri
puts bill.value
puts bill.to_date
puts bill.taxes

If your model was ready to accept such xml, Restfulie will do the job, whilst if it doesn’t recognize the attribute, it will still be available to you.

That’s the default Restfulie behaviour: to allow the other part to evolve their dynamic contract (and even parts of the fixed one) by default, without any extra effort from your side.

RESTEasy: Where did the hypermedia go to?

Some friends have asked what are the major differences between Restfulie and RESTEasy client frameworks.

Strong coupling and hypermedia awareless

As of today, Resteasy requires you to create an interface mapping every resource operation to a specific method, using @VerbName and @Path annotations to specify the desired target URI.

RESTEasy is ignoring the power of hypermedia content on the client side, thus being level 2 according to Leonard Richardson, and not matching Roy Fielding’s description.

RESTEasy is not Roy’s REST

According to Roy Fielding:

But hypertext as the engine of hypermedia state is also about late binding of application alternatives that guide the client through whatever it is that we are trying to provide as a service.

And any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type

Guess what? Java interfaces with @VerbName + @Path annotations means early binding, means an effort trying to describe which methods to use on which URI’s: tighter coupling, less space to evolve your server systems.

Going further:

It is fundamental to the goal of removing all coupling aside from the standardized data formats and the initial bookmark URI.

Restfulie provides a way to access only the initial bookmark URI, everything else is derived from navigating through your resources and its states.

Roy’s dissertation puts his believe quite clear in hypermedia being an important point of the web in chapter 5, his first paragraph: “Representational State Transfer (REST) architectural style for distributed hypermedia systems”.

RESTEasy got it right when it allows the user to access multiple URIs and the http protocol as the application protocol. But it still helps creating strongly coupled systems. Those who use JAXB (JAX-RS required) and fixed xml schemas do not allow server to evolve without prior notice to their clients. Ian Robinson has written about links in his blog and – another option – contract based systems in ThoughtWorks Anthology.

Back in 2007, there was a nice discussion on SOA allowing the users to create strongly coupled systems, where Mark Baker cleared it: SOA is constraintless, therefore one can create strongly and loosely coupled systems.

RESTEasy not supporting hypermedia content by default matches Roy’s definition of unrestful:

In other words, not RESTful at all. REST without the hypertext constraint is like pipe-and-filter without the pipes: completely useless because it no longer induces any interesting properties.

Although RESTEasy supports atom feeds, which is a good point, if the client wants a real hypemedia aware resource, he has to work on the deserialization process on his own.

RESTEasy does not provide the class generation mechanism that Restfulie (both Java and Ruby) provides so end users are able to transverse your resource’s tree and states without prior knowledge of every other URI.

According to Roy, one of the key benefits of the REST archicture style is its visibility through out the entire layered system:

REST is an architectural style that, when followed, allows components to carry out their functions in a way that maximizes the most important architectural properties of a multi-organizational, network-based information system. In particular, it maximizes the growth of identified information within that system, which increases the utility of the system as a whole.

By fixing all URI’s on the client side, along with not supporting hypermedia content by default, RESTEasy provides less information to layers contained between the server and the client: thus less visibility.

Server side

The same issues arrive when talking about the server side: RESTEasy does not provide any default interface for hypermedia content, but JAX-RS default support to JAXB: plain xml. It is up to (good) programmers to know how to make use of hypermedia systems in order to create a loosely coupled system.

Languages

Another key point on Restfulie’s approach is that it is Java-INdependent. Every month we see a larger number of integration software being developed in other languages: Restfulie can already help Java and Ruby developers.

Moving URIs around: how careful should I be?

Entry Points

In the human web, the entry point is a strong coupled URI.
Changing your company’s entry point from www.caelum.com.br to www.newcaelum.com.br without notifying your
clients will cause serious damages to those using your website.

The same holds for REST applications. Entry points to your system are strong coupled and
changing it without notifying your clients will break their system usage.

Entry point’s should not change without noticing your clients.

How can my clients become aware of the URI change?

Once you have published your new URI, set your old URI response to “301 – Moved Permanently” and set the header “Location” with
the appropriate location. Once 301 is received your clients can cache this response forever, changing their entry point configuration, because the move is permament.

By adding a proxy server to your system, the Moved Permanently response will be cached.
Restfulie will automatically redirect the user to the new URI upon receiving 301 on GET and HEAD:

product = Product.from_web 'http://www.caelum.com.br/product/2'
# received a 301, went to the new Location

If the request is anything but GET or HEAD, the user has to explicetely define its intention on following 301:

# explicitely following 301 on other verbs
class Product
	uses_restfulie
	entry_point_for.create.at 'http://www.caelum.com.br/product'

	follows.moved_permanently
end

product = Product.remote_create Product.new
# received a 301, re-posted it to the new uri, received a 200

Internal elements

In the ideal human web, all gettable resources are bookmarkable therefore any changes to those resources also
imply possible damage to your clients. After accessing www.caelum.com.br for the first time, if I access a ruby training,
I get to www.caelum.com.br/cursos/rr-71. I can use this address in several ways, i.e. bookmarking or sharing it.
If the courses URI’s changes, bookmarked URI’s will not work: the user will only find it out after trying it.

Once he has got a 404, the user will go back to the entry point and restart his procedure to
access the previous (invalid) bookmarked resource if the server does not answer with a Moved permantently response.

That’s what we do when our bookmarks break.

Again, the same holds for REST applications. In order to keep the benefits form Addressability, once published, one has
to keep his URI’s valid forever: either responding with its content, a See Other (303) or a Moved Permanently (301) response.

REST bases itself in addressability and therefore it is ok to bookmark any resource URI in the client system:
if you believe some your resources will not be bookmarked, it is ok to move them without prior notice, but remember, by
thinking that your resources were not bookmarked, you have accepted to lose one of REST’s benefits.

Client’s can always go back to the entry point and navigate through the previous executed procedure to retrieve the new URI,
but if he has to, it means that you broke one of your clients expectations.

302 and 303

Two other possible responses are 302 Found and 303 See Other.

While 302 means a temporary move, the client not supposed to cache the Location header as the new location
for the resource unless explicitely specified. Restfulie deals with it the same way as it does for 301:
you should notify Restfulie to follow 302 responses for verbs other than GET and HEAD>

When receiving a 303, the user should retrieve the resource at the new location with a GET verb.
This response allows the server to respond to a POST request by notifying the client to redirect to
the new resource URI – or any other desired one – with a GET.
The 303 response itself must not be cached.

Restfulie follows all 303 responses by default.

Summing up

No matter believing or not that your client will not cache your internal URI’s, the HTTP application protocol allows him to do it,
so in order to allow your clients to evolve appart from your own evolution, do not change the addresses
without keeping a proper Moved Permanently (301) response at those URI’s.

Internal element’s patterns might not be shared, but remember: every published resource should answer properly if they move.

Restfulie Java: quit pretending, start using the web for real

Its time to release Restfulie Java, offering the same power encountered in its ruby release, through the use of dynamic bytecode generation and request interception using VRaptor.

Serialization framework

Restfulie adopts XStream by default. Its simple usage and configuration gets even easier due to vraptor’s serialization extension built upon XStream – but it allows the usage of other serializers.

The following code will serialize the order object including its children items (much similar to rails to_xml only option):

serializer.from(order).include("items").serialize();

Connected: Hypermedia content creation

In order to guide the client from one application state to another, the server needs to create and dispatch links that can be interpreted by the client machine, thus the need for generating hypermedia aware serialization tools and consumer apis.

Its the basic usage of the web in a software-to-software communication level.

Pretty much like Restfulie’s ruby implementation, by implementing the getFollowingTransitions method, restfulie will serialize your
resource, generating its representation with hypermedia links:

public List getFollowingTransitions(Restfulie control) {
  if (status.equals("unpaid")) {
    control.transition(OrderingController.class).cancel(this);
    control.transition(OrderingController.class).pay(this,null);
  }
  if(status.equals("paid")) {
    control.transition(OrderingController.class).retrieve(this);
  }
  return control.getTransitions();
}

Controller interception

Restfulie for Java goes further, intercepting transition invocations and checking for its status. The following example will only be executed if order is in a valid state for paying:

@Post @Path("/order/{order.id}/pay")
@Consumes("application/xml")
@Transition
public void pay(Order order, Payment payment) {
	order = database.getOrder(order.getId());
	order.pay(payment);
	result.use(xml()).from(order.getReceipt()).serialize();
}

Why?

Restfulie does not provide a bloated solution with huge api’s, trying to solve every other problem in the world. According to Richardson Maturity Model, systems are called RESTFul when they support this kind of state flow transition through hypermedia content contained within resources representations:

<order>
 <product>basic rails course</product>
 <product>RESTful training</product>
 <atom:link rel="refresh" href="http://www.caelum.com.br/orders/1" xmlns:atom="http://www.w3.org/2005/Atom"/>
 <atom:link rel="pay" href="http://www.caelum.com.br/orders/1/pay" xmlns:atom="http://www.w3.org/2005/Atom"/>
 <atom:link rel="cancel" href="http://www.caelum.com.br/orders/1" xmlns:atom="http://www.w3.org/2005/Atom"/>
</order>

Stateless state

In order to profit even more from the web infrastructure, Restfulie for Java provides a (client state) stateless api.

Addressability

VRaptor’s controller api allows you to specify custom URI’s (and http verbs) to identify resources (and transitions) in your system.

Addressability + client cache stateless state server allows one to achieve REST’s idea on cache usage and its related layered systems advantages by allowing other layers to be added between the client and the server.

Unknown usage of my resources

Addressability + hypermedia content allows clients to use your resources pretty much in a way that was not tought of at first. Addresses (in our case, URI’s) can be passed around from one application to another, to and from a client’s internal database (as simple as a browser favorites, or google gears).

Building your system upon such basis, it become unaware of its resources usage (resource representation’s interpretation) patterns, allowing clients to create such previously unknown systems.

Less and simpler code, more results

Both on the server and client side, restfulie tries to achieve results based on conventions, therefore one can easily access its entry api to insert a resource in a system, i.e.:

  Movie solino = new Movie();
  resource("http://www.caelum.com.br/movies").include("metadata").post(solino);

And after creating a resource, one can actually navigate through your resource’s connections:

  Movie solino = resource("http://www.caelum.com.br/movies/solino").get();
  Comments comments = resource(solino).getTransition("comments").executeAndRetrieve();
  // print all comments

And navigate through your resource’s states:

  Comments comments = resource(solino).getTransition("comments").executeAndRetrieve();

  Comment comment = new Comment("nice movie on generations of immigrants and the hard times that they face when moving to a new place");
  resource(comments).getTransition("add").execute(comment);

A lot of good practices should be put into place in order to avoid creating Leonard Richardson’s defined REST-RPC alike systems.

Those good practices involve simple steps as avoiding anemic models on the client side . Restfulie+Vraptor already tries to avoid this on the server side andwe will discuss about such practices in following posts.

Download and example applications

Beginners could start by downloading restfulie’s client and server example application, ready to run in a eclipse wtp enviroment or pure eclipse installation.

Users are encouraged to use either the java or ruby mailing lists.

Since Restfulie was born in Ruby…

Since we released restfulie for ruby (on rails), which can be found at its github page, it was commented by Jim Webbers both on his blog and in person during QCon in San Francisco, where both him and Ian Robinson held a tutorial on restful systems and the web and will hold another round, more hands-on, at QCon London 2010.

Meanwhile, Restfulie was commented at ruby5’s podcast, commented here, at infoq, and in portuguese by Juliana and by Anderson Leite – a fellow Caelum developer.

As some comments were out about restfulie’s ruby implementation, restfulie: it’s more than easy.

pulling all my git projects at once

This is a short post for those who, like me, work offline at several projects during your weekend.

We have released a small command line tool that helps you sync all your master branches at once, by just running one command, all your repositories will be pulled again (and come back to your current branch, no automatic rebase)

scmall

More info at scmall website.

You can configure the files to work stashing data or even with svn if you wish.

missing_knowledge: dynamically adding ruby methods

While creating restfulie we have come across a few issues related to my complete lack of mastery of the ruby meta programming features.

The client’s code (hypermedia aware resource consumer) is based on ruby’s open classes and objects idea: while parsing a xml (or json) format, one can easily add specific methods to that particular object, and that is no news for anyone using ruby, but the interesting bit is that one can do it in several ways, each one with a complete different result.

statically defining methods

The first thought is to define a method for that just deserialized object:

object = self.from_xml
def object.pay
  // execute payment request here
end

This is the easiest approach and works really well if you know which method you want to define. In our case, the name of that method only become available during runtime and it is contained within an string so I can not execute:

object = self.from_xml
method_name = "pay"
def object.method_name
  // execute payment request here
end

In order to dynamically add methods to objects, the simple “def variable_name.method_name … end” path does not sound as a viable alternative.

missing a method?

The second approach was to use the method missing approach. By iterating over the possible state changes/requests of a resource, one can define whether a new method exists or not:

# keeps track of all possible state
# changes and requests
attr_accessor :state_changes
def method_missing(m, *args, &block)
  if state_changes.keys.include? m
    # execute the related http request
  else
    super(m, args, block)
  end
end

Whenever one redefines method_missing, he has to redefine the respond_to? method:

attr_accessor :state_changes
def method_missing(m, *args, &block)
  if state_changes.keys.include? m
    # execute the related http request
  else
    super(m, args, block)
  end
end

def respond_to?(m)
  state_changes.keys.include?(m) || super.respond_to?(m)
end

This is a great (and possibly only) approach to define new methods unknown to the developer at compile time which name is a composition of important strings as with ActiveRecord finders.

The keyword here is new. That’s the reason why method_missings is not the right approach for apis like restfulie. While retrieving a resource from the web, one of its actions might be named destroy, in which case the original destroy method will be invoked, without ever passing through the method_missing once the method does exist.

Appart from that, it is important to notice that this approach is highly valuable for those frameworks which supply dynamic method names for their users as mentioned earlier as ActiveRecord: in those cases, an infinite number of methods is made available for developers.

dynamically defining a method

Become more and more hardcore in the way to define a method, the following step was to check the define_method method. It seems to fit as it would receive a string as the method name and add that method to the object’s class.

class << self
  define_method("pay") do |payment|
    # execute payment here
  end
end

There is one issue here: define_method receives a block as an argument, and blocks cannot receive blocks as arguments in ruby 1.8. It is fine in 1.9, but not in 1.8, so I can not do:

class << self
  define_method("pay") do |payment, &block|
    # execute payment here
  end
end

Another important characteristic of using blocks with the define method, is that it can not access its local context (outter variables), unless you one uses the send method to invoke define_method:

self.class.send :define_method, pay do |payment|
  # access external variables
  # but no body support
end

module eval

The last approach is based on method aliasing. One can evaluate a block of code in a module using the module_eval method:

self.module_eval do
  def fixed_name_method(*args, &block)
    # now i can access the args
    # and the block
    # but the method name was fixed!
  end
end

So we are left with only one task: rename that method from fixed_name to the content of the variable method_name:

method_name = "pay"
self.module_eval do
  def fixed_name_method(*args, &block)
    # now i can access the args
    # and the block
  end
  alias_method method_name, :fixed_name
  undef :fixed_name
end

Note that we also undefined the fixed_name method after creating our desired alias. This is a known path to dynamically add methods to classes bypassing ruby’s 1.8 limitation with blocks receiving blocks as arguments.

Everything would be wonderful just prior to using the api in a multithread enviroment. In this case all those method definitions were added to all existing objects of that class. In this case, the use of instance_eval is indicated.

current method name

At last we come to the last challenge for the client side: once all methods are defined in the same way (within a loop condition) where the evaluation process does not give us access to outter variables, how come we know which method was invoked?

Googling around, the following method can be created in order to discover which method is being invoked:

def current_method
  caller[0]=~/`(.*?)'/
  $1
end

the next step

This is the path we went while creating restfulie as I had no prior hardcore experience with ruby’s metaprogramming techniques. It is clear that using such api allows one to dynamically extend classes and objects in order to achieve that “one extra dimension” of generalization to our code.

quit pretending, use the web for real: restfulie

Resource representation

A simple resource representation in xml would be as

<order>
  <product>basic rails course</product>
  <product>RESTful training</product>
  <price>500.00</price>
</order>

Note that it provides information on the order itself, as we are used to. But it lacks providing meta information on the business process: from here, I am unable to go anywhere.

This resource still lacks a way of notifying the consumer what are the available steps further on my business flow, i.e. paying this order or canceling it.

Looking at this resource, it is clear that it is just an usual “structural representation” of something that programmers would code in C: the data and it’s structure: but no possible actions.

Identifying resources

In order to access those resources, one needs to possess its address, its URI.

We use, for example, URIs to identify our courses at Caelum.

The advanced rails training can be accessed through the http://www.caelum.com.br/curso/rr-75-ruby-rails-avancado/ URI and also through http://www.caelum.com.br/curso/rr-75.

Note that there is no need to have an unique URI to resource or resource to URI mapping. In our case, the latter URI redirects to the first one through an http permanently moved response.

Web services

One can understand web services as services which use the web as its infrastructure. The basic step to create a web service is to use http requests to act.

Typical SOA applications will stop its evolution and its use of the web infrastructure here, not profiting from everything else that the web provides.

Resources accessed through the web

Once I have an URI in my hands, and a web service that provides me the resource representation through a web request (typically a GET request), one receives as output the requested resource representation.

Using different HTTP verbs and analyzing the request and result headers and codes, one makes and profits from further usage of the web infrastructures such as caching proxies and fault tolerance assurance due to idempotent methods.

Frameworks and applications, like the one from the spring blog, which provides the first level of web infrastructure usage would be called resource based web services (RBWS) if they were not lacking the use of the most common web aspect: hypermedia.

According to Roy Fieldings, “REST is defined by four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state.”

Web service based CRUD

Web service based cruds are those ressemblingthe spring tutorial, which maps URIs to resources and allows the usage of different HTTP Verb, headers and return codes to sign actions and return values.

Although being a sign of better usage of the web in order to create web-based services, it is still missing the very best part of our every day life: hyperlinks and hypermedia content.

Doing CRUDs is an everyday task, and doing service based CRUD implementation is a nice and cute functionality that frameworks as rails, spring and vraptor allows us to do.

According to Leonard Richardson model, one can roughly categorize his usage of the web infrastructure based on which use one makes out of it. CRUD over the web is just the first step, and hypermedia support is the following one.

Hypermedia aware resources

Resources supporting hypermedia provide consumers the next steps to follow on an application business flow. It allows the consumer to change the application state as web requests are stateless.

The following code shows an example of xml+hypermedia representation of the same order seen earlier:

	<order>
		<product>basic rails course</product>
		<product>RESTful training</product>
		<refresh>http://www.caelum.com.br/orders/1</refresh>
		<update>http://www.caelum.com.br/orders/1</update>
		<pay>http://www.caelum.com.br/orders/1/pay</pay>
		<destroy>http://www.caelum.com.br/orders/1</destroy>
	</order>

Note that this path is a simple yet powerful way of interacting with your clients. Although its a valid option, one can make use of the w3c atom format as nowadays there are hundreds of important tools atom-aware:

<order>
  	<product>basic rails course</product>
  	<product>RESTful training</product>
  	<atom:link rel="refresh"
          href="http://www.caelum.com.br/orders/1"
          xmlns:atom="http://www.w3.org/2005/Atom"/>
  	<atom:link rel="update"
          href="http://www.caelum.com.br/orders/1"
          xmlns:atom="..."/>
  	<atom:link rel="pay"
          href="http://www.caelum.com.br/orders/1/pay"
          xmlns:atom="..."/>
  	<atom:link rel="destroy"
          href="http://www.caelum.com.br/orders/1
          xmlns:atom="..."/>
</order>

Cutting a new gem

Both Jim and Savas have commented on the client simple javascript+html resource+hypermedia (restful?) client that we have created so one could easily test a restful application online.

So me and Caue Guerra at Caelum were ready to move to the next step: getting resource+hypermedia support to rails applications.

SOA applications, due to strange and dark forces, typically demand dozens of developers to built something complex and hard to maintain.

With this in mind, we have developed Restfulie, which is a rails based gem.

It works both on the server and client side making it easier to provide and consume such restful resources.

Requesting a resource

A resource based web services provider – another long name for a partial rest implementation based on roy’s dissertation – can access a resource through the api:

 order = Order.from_web ‘http://www.caelum.com.br/orders/1’

Now, one can request its content

 puts order.products

or even invoke those “methods”:

 order.pay payment
 order.cancel
 payment = order.check_payment_info

All those three methods are remote invocations (never forget it!) that access the server executing POST, DELETE, PUT and GET operations, serializing and deserializing xml+atom content (or json, …).

Thats about it. You have consumed a resource representation and acted upon it in two lines of code, moving your resource to another state on your business flow.

Meanwhile, on the server side

Those who want to provider RBWS can easily do it using restfulie by implementing the method following_states as follow:

class Order
 def following_states
 states = [ { : orders, :action => :show } ]
 states << : orders, :action => :destroy} if can_cancel?
 states << : orders, :action => :pay, :id => id} if can_pay?
 states
 end
end

And in a 6 lines long method, one has implemented a full RBWS provider: all invocations to to_xml and to_json will serialize atom or rel based links to those state transition services.

Live examples

You can try the server side ordering application online and a sample client side application live at heroku.

In order to fully test those systems, follow the “Try it online” directions on the Restfulie website.

Concluding

This post gives a basic idea on resources, web based services and how one can really use the web to profit from its already existing infrastructure.

While some RBWS providers already use all this knowledge, most of the frameworks that we see today focus on “restful” meaning “accessing a resource through its URI and http verbs”. There is more to the web than just getting, posting and uris. There is a whole world hidden behind links so… let’s quit pretending and start using the web infrastructure for real.