Are ORMs Solving Anything?

Angry Baby

I like to write blog posts where I offer some pragmatic advice.  In most posts I try to include tons of code samples and example projects...but this post is different.  I am trying to get my head around something so I want to share what is in my head so I can get a conversation started with my readers to help me out.  Once you read this post, please comment...

The other day I was responding to a tweet from Doug Purdy. He had posted a link to some new EF 4.0 features by the boss at Microsoft's DevDiv. I, as usual, complained instead of lauded the list. I started a conversation about lazy loading and the potential danger of it, but quickly Doug mentioned that ORM's maybe were the wrong approach in general. That got me thinking (not always a good idea).

Those of you who have followed me for a long time are apt to remember that I go back to ADO (actually even farther back) to a time when writing your own data access layer was the norm. And using that data access layer against your business objects was what we all did.

Things are really different today as most of us now use an Object Relational Mapper (ORM) of one sort or another.  Entity Framework, LINQ to SQL, nHibernate, LLBGenPro, etc. They all essentially do the same thing (but in very different ways). The problem is that to some it feels like ORMs are business objects. In fact, ORMs are just data access. Are they making our jobs easier?  It depends...

Most ORMs lean heavily on code generation to create entities and other code. The mapping is generally trying to take some relational model and create an object oriented model. It works this way because this is what we've done for a couple of decades.  But is it helpful? Perhaps not. But because we build class-based (or object oriented) software today that seems plausable to do. 

Object orientation is at a cross-roads in many ways. We've been told for a couple of decades now that it is the way that you can benefit from reuse, ease of development and modeling. Of course, like all magic pills, applying it to every problem space doesn't help. In the case of ORM's, its always felt like we were putting a round peg into a square box.

The StackDon't get me wrong, mapping a rectagular result (e.g. SELECT * FROM Foo) makes a lot of sense.  But the problem is that the mapping happens between *related* entities which is hard. Relationships in data stores are not as simple as 1..n, 1..1, or 0..1. By shaping the relational model into a class structure, we're losing fidelity.  You can see if you're losing fidelity when an ORM is going to allow you to get deep inside to hint at the actual queries. At that point we're fighting the tools.

I am not anti-ORM at all and I can clearly see how ORMs help build smaller projects easier.  This is especially true if your relational model looks like the object model (or as is common in DDD, using the class model to push down the schema). Great if you're building small projects, but when those same techniques are used on large or enterprise solutions, the stack gets big fast (see right).  The problem is that we end up with in-memory classes that represent entities, then we need business objects that apply business rules, then DTOs to communicate the shape of hte data across the wire and in some cases data contract classes outside the firewall. Under the ORM entity objects, the ADO.NET managed providers still exist and sometimes an unmanaged stack too. Sometimes that a lot of code to worry about.

My big worry here is that ORMs are seductive.  When you start a project and build a data layer with a couple of clicks you can focus on building the benefit to the business.  Like most development issues, the real cost is in the maintenance. I fear that much of the work in this space is done by consultants because typically consultants help get a solution built and move on. Its not that consultants are bad, but because they can go from greenfield to greenfield project (I certainly have been guilty of this before), the pain of maintaining this code isn't always obvious.  What happens when the database is overwhelmed with data.  What happens when you try to add indexing to an existing application as the use cases change and mean that adding index hints is hard (though in SQL Server 2008 you can do this without modifying the calls).

That comes back to Doug Purdy and other folks in the twitterverse.  What's the next wave of solutions out there?  I admittedly haven't spent enough time playing with solutions like ActiveRecord, but they feel like dynamic classes which doesn't feel like it will solve the problem. I have heard that maybe getting away from relational databases is the solution, but the object database/non-SQL/BigTable solutions I've seen never seem to scale to high transaction systems or work well with reporting.

Your turn...what's your take?

Comments:

Gravatar

Like most technologies, I have found ORMs to be useful, but not nearly as useful and capable as perhaps advertised and hyped.

As soon as you have written a few applications and find yourself writing the same data access code and having to update it in multiple places, ORMs become appealing.

Of course, they add their own additional work and as noted, can fall short when data models become more complex.

For my money, ORMs are far from perfect, but do provide value.

Gravatar

I agree, when your model matches pretty close to your DB then ORM's are valuable. However, for Larger more complex projects ORM's tend to get in the way and cause more problems than they solve.

On a similiar note, do DI Frameworks really solve anything?

Gravatar

Perhaps it was missed in the prose, but to be honest I am not anti-ORM, I am just looking at what is coming to solve these deficiencies that I haven't noticed.

Gravatar

IMHO ORMs and the poor man's ORM (hand coded or generated DAL) is a fact of life unless you are willing to forgo using OOP in a system that gets its object data from a RDBMS. There is just simply no way around it. Even active record requires an ORM unless you are willing to pass Datasets around.

Gravatar

I've struggled with this problem, for years. Like you mention, we could move to an object database, but they
have their own set of problems. I have found no easy solution for this problem other than trying to keep my code as loosely coupled from the data model as I can get it. This keeps me from making massive code changes if the model changes, for example.

If there is a solution, I have to imagine it's going to take a completely new way of thinking. Many of us have become so accustomed to the way it had been done for years that we are no longer able to think outside the box.

Gravatar

I am glad that we don't have to figure this out on our own. The java community as well as others have already vetted the ORM solutions as fit for a wide variety of apps. I look to others who have had to make similar decisions. That said, .net had an orm in .net 1.0. SqlcommandBuilder. However crude, it made it so that you didn't have to write SQL. It just didn't support rich objects.

Gravatar

Hi Shaun,

This is certainly a thought provoking post. I wrote a rather lengthy comment which was promptly rejected due to length, so I've responded instead on my nascent blog:

http://blog.adamjcooper.com/2010/01/are-orms-solving-anything-yes.html

I'd love to hear your thoughts.

Take care,
Adam

Gravatar

Like most technologies, I have found ORMs to be useful, but not nearly as useful and capable as perhaps advertised and hyped.

As soon as you have written a few applications and find yourself writing the same data access code and having to update it in multiple places, ORMs become appealing.

Of course, they add their own additional work and as noted, can fall short when data models become more complex.

For my money, ORMs are far from perfect, but do provide value.

Gravatar

Hi Shawn,

I can see a lot of ways to extend the traditional "ORM" approach in a more enterprise-aware manner. But I think this is like saying "We'll make the Titanic stronger and more unsinkable than any other ship in history!" Sadly, a bridge officer who had a few too many vodkas will still find a way to sink the vessel.

Perhaps what we need in our industry is an even more broad-reaching conversation on data architecture itself?! What does data look like in 5 years time? 10 years time? 50 years time? Given the enormous abundance of compute cycles, storage and system bandwidth that will be at our disposal, is normalization even worth anything any more? Does the cost of looking up a surrogate integer ID "primary" key justify the disconnection of the key from its supporting data when that "cost" is infinitessimal in comparison to the capabilities of the hardware? What new ways of organizing information will be made possible by this explosion in compute capacity?

I think these are the conversations that are necessary and overdue; and are way more important than Ted Neward's "Vietnam of Data". I think the emotion that goes into ORM debates along the lines of "Communism sucks"/"So does capitalism" suggests that we need a "New World Order" that transcends the existing debate and starts focusing us as IT professionals in an entirely new direction.

Cheers

--
@Ozziemedes
http://ozziemedessql.blogspot.com
http://ozziemedes.blogspot.com

Gravatar

Shawn it's good to do thought provoking post once in a while. So what do you suggest ? Criticism is great, but if it's not constructing anything, it's called a rant.

So do you suggest building SQL and then business objects on it and then when things change(it happens far too often), do the changes at SQL, DAL, Business Layer, UI and also your tests ?

ORMS are a reality and unless some document oriented database takes off, it's only way of development in complex projects.

Gravatar

Hi Shawn,

I can see a lot of ways to extend the traditional "ORM" approach in a more enterprise-aware manner. But I think this is like saying "We'll make the Titanic stronger and more unsinkable than any other ship in history!" Sadly, a bridge officer who had a few too many vodkas will still find a way to sink the vessel.

Perhaps what we need in our industry is an even more broad-reaching conversation on data architecture itself?! What does data look like in 5 years time? 10 years time? 50 years time? Given the enormous abundance of compute cycles, storage and system bandwidth that will be at our disposal, is normalization even worth anything any more? Does the cost of looking up a surrogate integer ID "primary" key justify the disconnection of the key from its supporting data when that "cost" is infinitessimal in comparison to the capabilities of the hardware? What new ways of organizing information will be made possible by this explosion in compute capacity?

I think these are the conversations that are necessary and overdue; and are way more important than Ted Neward's "Vietnam of Data". I think the emotion that goes into ORM debates along the lines of "Communism sucks"/"So does capitalism" suggests that we need a "New World Order" that transcends the existing debate and starts focusing us as IT professionals in an entirely new direction.

Cheers

--
@Ozziemedes
http://ozziemedessql.blogspot.com
http://ozziemedes.blogspot.com

Gravatar

'ORM' (bad acronym btw, ORM == Object Role Modelling', is about entity services, not about fetching entity instances into class instances from databases. Entity services are about persisting graphs of objects with their entity instances, making them stay in sync, etc. etc., not about being able to run a handwritten query to save a row somewhere.

The advantages of O/R mappers lay in the area of entity services, where any other entity manipulation code (or if you will, 'data access code') will fail you as it lacks any entity services. So are they solving anything? Yes, they provide you with services you otherwise would have to write yourself. That's what they do, nothing more, nothing less. It's not about fetching data, it's about the functionality on top of that.

Gravatar

Even if you don't use the ORM generated classes as your business objects, you can still use anonyous type queries. Common problems with fully using ORM classes as the business objects are pulling back too much data (all fields on any objects), and lazy loading complexity (I want this child but not this other child).
With LINQ to XXX you don't have to write any SQL, it is all typesafe, and you can get related child data easily. So, with anonymous type queries you are still interacting with the database in an object oriented fashion, and controlling specifically what columns of data you receive.
Removing the ORM and LINQ and going back to writing SQL with datasets, you can see that it is a large step away from ease of development.
Maybe loading your own business objects using anonymous types can be a middle ground.
Hopefully what I tried to describe made sense :)

Gravatar

Shawn:

To be honest, I have not yet played enough with EF in 4.0 to make a decision. The POCO bits sound like we can actually get business objects. Until I play, I am not sure.

I do agree, however, on a couple of points:

1. The drag and drop stuff often makes initial development easy at the expense of maintainability, which ultimately costs companies a lot of money. Yes, I am a consultant that actually gets this. ;-)

2. That we often box ourselves in with the drag and drop seductiveness.

Most of the tech you mention forces you to create some form of "translation layer" if you want to be able to swap things out, as the "objects" created are not really compatible. I am a firm believer you should create your own business objects, so this is not an issue for me, but I hear people saying "it is easy to move from DataSets to LINQ". Yes, but not in a single project.

Peace and Grace,
Greg

Twitter: @gbworld
Blog: http://gregorybeamer.spaces.live.com

Gravatar

It's hard to imagine moving away from the relational database model to store data. Even with an ideal object-based solution in place to access the data, I'm still going to want a way to "directly access" my data without having to instantiate objects to do so. If relational databases are here to stay, the best solutions we can hope for are better ways to interact with that data from a development standpoint. That's where ORMs step in to try to help.

As with any new layer of abstraction (in this case the object-layer that's intended to help us "get at" the data in the database), there's no perfect solution, but each attempted solution will bring us closer to the ideal. For example, if you've tried to roll your own data access layer before, you know a lot of the challenges and pitfalls with doing so. When you try out the latest and greatest ORM, you'll know what pitfalls to watch for and how far you'll be able to scale it (hopefuly *before* you start pulling your hair out).

Perhaps some of the better data access solutions will crystalize for certain frameworks over the next decade, but as you pointed out, they won't always scale to every situation. I guess that means you'll still have a job helping people figure out their data :-).

Gravatar

I just read my result set into a DataTable and pass it back. To me ORMs just cause more problems.

Gravatar

I certainly agree with the others here, this is a problem we have had for as long as we've been using RDBMS as the persistence part of OO Designs.

Unfortunately attempts to take the RDBMS out of the equation have failed, they're just too good for other stuff we need for that.

Ultimately I consider ORMs helpful, they take the leg work out of some coding. However as with all tools there is scope for abuse. Many tools allow generated "Entities" to bound directly to UI elements so very, very easily.

Gravatar

I think the real problem is RDBMS is not the data store we are looking for, and we are all just dealing with the kludge until we find that right store.

Gravatar

I think the ORM is just a tool to help us handle db access better. However, as with every wrapper, some of the original functionality is changed or lost.
We can now choose to create a better ORM or stop using it at all, according to our needs.
I am using Subversion for an extensive amount of time, and I really hate that tool. It just won't do what it should do, and does all kinds of other things that it shouldn't. However, I am still using it, because I need version control (but I feel like I should switch to another versioning system)

Gravatar

I think ORMs are useful on projects where the budget/team is small, time is tight and the client wants to see results yesterday, i.e. a 'RAD' environment.

Conversely, in a large enterprise architecture where there are processes like compliance, SDLC and an obsession with code-reuse (by that I mean code written 8 years ago which management does not want obsoleted and any new technology less than 4 years old is shunned - I'm sure we've all worked there) then ORMs get less playing time.

With proper separation of concerns in your code, you should be able to use ORMs as and when with little difficulty.

Gravatar

From a large-scale, enterprise solution perspective I think the key is to look at the ORM generated code strictly as data access/data manipulation code. It frees the developer from writing mundane CRUD code over and over. If you never have to write an INSERT, UPDATE or simpleton SELECT for a given system, then the ORM has added some value. As long as RDBMSs are in play, "objectifying" the data access has some benefit to the developers.

I think one of the big problems is that many of the ORMs are now migrating towards the Swiss Army Knife end of the scale and their story is getting kind of fuzzy. Hoping and believing that these frameworks will somehow replace well thought out, custom domain specific objects is a lot like drag-and-drop programming. It demos well and holds lots of promise but never quite gets you to where you need to go.

twitter: @mjmo


Gravatar

Certainly - O/R mappers do not solve every problem, and, like a lot of other technologies, they're often blindly applied (see Uncle Bob's recent post on Dependancy Injection for an example).
However, if you practice Domain Driven Design, and try to apply SOLID principals, then O/R mappers just seem to fit.
That said, I've mainly used NHibernate, (and I've been doing that since version 0.4) - the lack of d&d designer means that I've had to think about every line of .hbm.xml I've every written (and can now celebrate over AutoMappings). This, with DI, means I can build a testable, flexible app. From experience, drag&drop development doesn't give you this, nor does code gen.
So yeah, O/R mappers do have a use, but (a) get to know your tools, and (b) don't forget about OO techniques.

Gravatar

“but when those same techniques are used on large or enterprise solutions, the stack gets big fast (see right).”

That would happen regardless of persistence mechanism if you chose that architecture. Mapping from an ORM Code -> Business Object -> DTO is a choice, not a requirement. No ORM requires this architecture or somehow leads you too it in a way a custom DAL does not. SOA leads you to this architecture. Losing object refs across the wire is the real problem.

Gravatar

Maintenance wide, having more control on the codes is better than depending on the automatically generated codes, especially in the scenario that the techniques are changing so fast and the generated library are getting heavier and heavier. I still prefer a slim DAL because though the ORM tools\libraries are getting bigger, they are still not quite satisfied.

Gravatar

LIke many others here I believe that ORMs have their time and place. Their time is when the schedule is very limited and you are trying to develop a prototype or small simple applications. The place is probably more in the consulting/small enterprise environment. Personally I have stayed away from automated ORM systems, I have looked into nHibernate and EF and in both cases I have maintenability to be an issue.
I have yet to come across a system where the database schema and model where so tightly coupled that minimal changes were required upon generation of the domain model. More times than not I find myself making changes to the data schema, and reflecting those changes in the domain model. Doing this with an ORM because a nightmare because it requires the regeneration of code.
Also, when I write an object model layer, I am not only providing data access to the UI, but there is significant embedded business logic. Using an ORM would signify the creation of another layer of abstraction where to codify the business rules. This seems to me like a waste of time...

twitter: edcabre

Gravatar

What about oslo?

Gravatar

I've experienced what you describe about ORM's first hand. We're building a large scale application using EF. Because we have like 200 entities, we needed generic code to handle CRUD operations against EF, ending up building part of the DAL ourselfs, on top of EF, which kind of defeats the purpose. The only benefit we have left is the mapping itself, so we can do need tricks there.

Great post. Thanks.

Gravatar

In my opinion ORM's for enterprise applications should become "one shot code generators". I think they should offer customization about the code they generate allowing a clean/easy code that could be maintained later manually. I'm scared of using any kind of "auto syncronize model" black box button... Lack of control is my main complaint with ORM's. Maybe I didn't find the rigth one for me yet...

Gravatar

I'd like to see better tooling for DDD that devs are building and a better separation of 'database tables' vs OO design

There is too much dependence on mapping to existing legacy databases and their setups that don't match the dev requirements for building good applications

Gravatar

My current project is a huge order system for a global enterprise with about 3000+ stores around the world, we use NHibernate very successfully.

The system started with Linq To Sql and failed miserably but NHibernate just takes all our scenarios and swallows them.

It's more about understanding ORM as a data access technique, handling the impedance miss match between the two models and have some experience in tweaking that to your advantage. You need to know the limitations of both and make sure you model them so you don't hit that limitation (both the db and the Objects). This is however true with handcrafted DAL's as well.

A handcrafted translation layer in this system would probably cost us tons of more money, work and grief. As Frans mentions, the services an ORM can provide is just to good to pass for enterprise solutions.

I would argue that good ORM's might not be good fit for smaller projects though. The threshold is often to high to benefit, if you don't have people on your team really experienced with the ORM you are about to use.

Gravatar

The simple fact that ORM tools and APIs have been produced is a testament to the mismatch between RDBMS and OOP. In my opinion, RDBMS should be used primarily for reporting functions and data that is harvested at appropriate points in it's lifecycle would be written to the RDBMS for that purpose.

I'm in favor of data storage for front-end user applications going to a document storage system (MongoDB/Filesystem/Redis), with deferred processing to perform longer term, immutable writes to an RDBMS for aggregated ad-hoc reporting and finally to OLAP for dimensional metrics.

So in my mind there are three storage incarnations for information; 1. Document storage and retrieval, 2. RDBMS for transactional logging/reporting and 3. OLAP for analysis/reporting. Trying to do all three of incranations using an RDBMS and using a plethora of APIs to square-peg-in-round-hole the data, is too complex, it will break, cause pain and be difficult to maintain.

Let RDBMS be simple, stop trying to make it a single source of truth. Data wants to be free. Liberate it, stop trying to make objects into relational tables. Stop designing objects to conform to relational ideals. Take the power away from these crazy people, and better, simpler, more useful software will be the result.


 



 
Save Cancel