Rigidity in Data Design

August 9, 2008
No Comments.

Silverlight Logo
To many developers this may seem odd. I talk with many staunch ALT.NET guys, and the DDD philosophy seems to be that data is a top-down or at worse, bottom up design problem. The issue here is that there is an assumption that just simply not true that data design is part of most software development projects. The reality based on my experience as well as the experience of talking with developers in the community is that many projects (though its hard to exactly quantify what percentage) begin with existing data. This is especially true in the enterprise where data exists in many forms from new databases, legacy servers (e.g. mainframes) or even flat files and XML. It is the rare project that is new code against all new data.

For new projects, creating a new database schema that at least in some part mimics the design of a project makes sense. The problem is that over time the data must evolve. I think the reality is that depending on the organization this can be very different. There are some opinions that I think impact the maturation of a data model:

  • Well if they want access to the data outside the application, I will expose that as a service interface.” - I think this is coming from the world of a consultant not a enterprise developer. The reality is that the data maturation happens over time with the help of DBAs, developers and IT people. Its a complicated situation and isolating the database to the non-development stake holders is something that just doesn’t happen in the long run.
  • Transactional schemas are not a good solution for reporting.” - While I don’t disagree with this idea, it doesn’t happen in the real work either. In most large organizations, they have need for data coming out of systems and cannot wait for data warehouses, data marts or reporting servers to be created and maintained.  If they are created as part of the application development, they usually get the short straw of maintenance.  Reporting happens against transactional schemas…period…get over it.
  • If the application is designed right, the DBA should never need to get involved…we can solve perf problems with caching and smarter data access.” - This assumes that the data is isolated from the rest of the organization.  This doesn’t happen either. In addition, data changes over time and that’s where DBA’s can be exceptionally helpful. In my opinion, the form of the data can be improved over time as real data is used in a system. What works for a 1GB database rarely works for a 1 TB database. Lastly, there are often many more readers of data than you might expect. This is related to the reporting story, but also ad hoc attaching of data (damn you Excel and Access) happens. You can’t control the gateway to the data as the guys that sign the checks sometimes make dumb decisions that impact the organization…that won’t change.

Now I am not suggesting that we need to dumb down designs to deal with the nature of data, but understand that the IT infrastructure of deployed products is a lot more fluid than developers would like. For me it is, “Hope for the best and plan for the worst.” This means understand that pushing a design philosophy that can help developers get their job done in a more efficient way as well as create better software is what we all are trying to do, but there are no black and white lines drawn here. No *rule* applies in all situations. I laud the ALT.NET guys for trying to inject better skills and tools into the process, I just rail against the ferocity of zealotry that comes in some of the message. When enterprise developers hear this, they just tune it out instead of taking what helps them and leaving the rest.

That’s just my opinion…