I recently recorded a couple of series of videos for MSDN's Data Dev Center to help people get up and running with OData. There are two series:
Getting Started with OData: This two part video walks through creating an OData (e.g. WCF Data Service) and consuming it in .NET.
Consuming OData: This six part series shows you how to consume OData on a variety of platforms including .NET, Windows Phone 7, Silverlight, jQuery, PHP and iOS (e.g. iPhone/iPad).
In this (somewhat belated) part 6 of my Architecture for the Windows Phone 7, I want to talk about dealing with data across the wire (or lack of wire I guess). If you've missed the past parts of the series, you you can visit them here:
This is at the heart of the idea that the phone is one of those screens in '3 screens and the cloud'. The use-cases for using data are varied including:
Consuming public data (e.g. displaying Netflix Queue or Amazon Catalog).
Consuming private data (e.g. showing your company's public data).
Data Entry on the phone.
When coming from Silverlight or the web, the real challenge is to meet your needs while realizing you're working with limitations. When you are creating an app for the desktop (e.g. browser, plug-in based or desktop client) you can make assumptions about network bandwidth and machine memory. While most developers won't admit it, we will often (to help the project get done) just consume the data we need without regard to these limitations. For the most part on the desktop this works as we often have enough bandwidth and lots of memory. On the phone this is definitely different.
You have a number of choices for gathering data across the wire(less) but the real job in architecting a solution is to get just enough data from the cloud. The limitations of a 3G (or eventually 4G) connections aside, making smart decisions about what to bring over is crucial. You may think that 3G should be enough to just get the data you want but don't forget that you need consume that data too.
I recently chagned updated my Training app for the Windows Phone 7 to optimize the experience. I found that over 3G (which is hard to test without a device) that the experience was less then perfect. When I built originally build the app, I just pulled down all the data for my public courses into the application. In doing that the start-up time was pretty awful. To address this, I purposely tuned the experience to make sure that I only loaded the data that I really needed. But what that meant at the time was to only pull down the information on the selected workshop when the user requested it. In fact, I did some slight-of-hand to load the outline and events of the workshop while the description of the workshop was shown to the user. For example:
void LoadWorkshopIntoView(int id)
{
// Get the right workshop
_workshop = App.CurrentApp.TheVM.Workshops
.Where(w => w.WorkshopId == id)
.FirstOrDefault();
// Make the Data Bind
DataContext = _workshop;
// Only load the data if the data is needed
if (_workshop != null && _workshop.WorkshopTopics.Count == 0)
{
// Load the data asynchronously while the user
// reads the description
App.CurrentApp.TheVM.LoadWorkshopAsync(_workshop);
}
}
I released the application to the marketplace and the performance was acceptable...but acceptable is not good enough. Upon refactoring the code, I realized that I was loading the entire entity from the server (even though there were a lot of fields I never used). It became clear that if the size of the payload were lower, then the performance could really be better.
This story does not depend on the nature of your data access. In my case I was using OData to consume the data. My original request looked like this:
var uri = string.Concat("/Events?$filter=Workshop/WorkshopId eq {0}",
"&$expand=EventLocation,Instructor,TrainingPartner");
var qryUri = string.Format(uri, workshop.WorkshopId);
By requesting the entire Workshop (and the related EventLocation, Instructor and TrainingPartner) I was retrieving a small number of very large objects. Viewing the requests in Fiddler told me that the size of these objects were pretty good:
HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 20033
Content-Type: application/atom+xml;charset=utf-8
Server: Microsoft-IIS/7.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 13 Nov 2010 21:17:38 GMT
By limiting the type of data to only fields I needed I thought I could see some small change. I decided to use the projection support in OData to only retrieve the objects I needed:
var uri = string.Concat("/Events?$filter=Workshop/WorkshopId eq {0}",
"&$expand=EventLocation,Instructor,TrainingPartner",
"&$select=EventId,EventDate,EventLocation/LocationName,",
"TrainingLanguage,Instructor/Name,TrainingPartner/Name,",
"TrainingPartner/InformationUrl");
var qryUri = string.Format(uri, workshop.WorkshopId);
Selecting these fields which were the *only* ones I used reduced the size to:
HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 8637
Content-Type: application/atom+xml;charset=utf-8
Server: Microsoft-IIS/7.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 13 Nov 2010 21:17:59 GMT
That's over a 55+% savings! The size difference wasn't only in bandwidth, but the cost in memory to deserialize the results, storage size in memory and CPU cost should be substantially cheaper too. I did the same thing with similar results to the Events list. I saw a 55+% savings there too.
While using OData's projection mechanism ($select) worked for my case, OData isn't special here. You could have done the same when you're building a web service or REST service. The only case where this type of decision isn't possible is when you're building an app on top of a service you don't control. Since most REST and Web Services don't have a built-in mechanism to limit the result set, you could proxy through your servers and trim the offending sized entities too.
The important thing to think about is that you're working with a different kind of platform than you have in the past (when working strictly on the desktop). You have to think about optimization the whole way through.
Now that the Windows Phone 7 Tools are fully released, I sit here waiting for the phones to be released. In the time between the tool release and the phones being released, I have some precious time to get my applications ready for release as well. While I am not building any mass-market applications (nope, no fart apps here); I am building a number of small applications that should show off some features of the phone.
While many of you have seen my Moon Phaser application, I wanted to do something that was more in line with a typical phone application so I built an app to review my company's training offerings. The goal was simple: allow users to see the classes we're giving and learn about the the classes (with hopes of signing up ;).
I had started this application before the the Windows Phone 7 Tools were complete so I started niavely building a panorama-type application:
(click image for full size)
The idea was to have a list of apps in the Workshop pane and once a workshop was chosen, the other three panes would fill with data. I got pretty far with this idea (not using the Panorama pane but instead using the Visual State Manager to move the page around). Once the tools were delivered I switched this to use a Panorama control instead. While this worked, the user experience was completely broken. If a user flicked right, there would be no data until you picked a workshop. It occurred to me that this design was completely off-base. I was trying to shoe-horn the Panorama style into my own application. I would have designed this type of app on the desktop as a 'explorer-like' interface (click on the left-hand list/tree view and show detail data). But that doesn't apply itself onto the panorama as much as you might think. The Panorama design I think works better for generalized hubs where you have different classes of functionality or data in different categories. One pane relying on the others is a broken user experience IMHO. Of course, who knows...we're early into the Metro style so I could be completely wrong.
In changing my application, I decided to go with page navigation and a pivot control. Essentially, the initial view of a workshop picker and then a second page with a pivot control for more information. I decided that the picker should be really simple with title, list of workshops and a logo:
Once the user picked a class, they could use a new page with a pivot to look at a mix of data:
I am pretty happy with the result. Here is a video of the app running on a device over WiFi:
My biggest challenge was to realize that this app was running on the phone. I know that seems obvious, but as a server, web and desktop developer I have learned that memory footprint isn't the most important thing (though it always is a consideration. For the phone this becomes a huge deal. In fact, according the the Applicaiton Certification Requirements (e.g. the rules of the Marketplace), your application can't take too much memory (emphasis mine):
5.2.5 Memory Consumption The application must not exceed 90 MB of RAM usage. However, on devices that have more than 256 MB of memory, an application can exceed 90 MB of RAM usage. The DeviceExtendedProperties class can be used to query the amount of memory on the device and modify the application behavior at runtime to take advantage of additional memory. For more information, see the DeviceExtendedProperties class in MSDN.
Of course the amount of data I was initially bringing down was not trivial. I was pulling down quite a lot of my object model on the server. Before I worried too much, I decided to take a look at the memory consuption. As the guidelines say, the DeviceExtendedProperties class is the trick to measuring this for your application. Since I was already using a placeholder for errors, I decided that under the debug settings I'd use a polling timer to show the current memory in megabytes to see what my consuption was:
You'll notice that I am calling DeviceExtendedProperties.GetValue to get the current memory usage of my application. There are values for application memory and application peak memory. The full set of extended properties are listed here:
You can see the memory usage shown in the application here (note, this won't make it to the release version):
On my initial checks of this, the memory consuption was huge (as I was pulling down all that information for all my workshops. This was causing a problem for both the bandwidth usage (e.g. networking speed) but also the memory consumption. I realized that users would likely not look at all the workshops so pre-loading all the data wasn't helpful. So I decided to split it up into three separate sections of data. But first, how was I getting that data you might be wondering?
To get the data, I decided to use my publically available OData feed (e.g. WCF Data Services). I decided to use this so I wouldn't have to have any server code at all. The public feed exposes all the data I needed so it should just work. The first step was figuring out how to consume OData on the phone. There is a CTP version of the OData WP7 library that exists on OData.org:
While the WP7 library works, it has a lot of limitations right now. The first thing you'll notice is that Visual Studio doesn't support "Add Service Reference..." so you'll have to build one using the command-line tooling as explained in my previous post here:
The other limitation you'll see is that the LINQ syntax isn't working right. So you'll have to execute your queries via the URI query syntax:
var qryUri = string.Format("/Events?$filter=Workshop/WorkshopId eq {0}",
workshop.WorkshopId);
Ctx.BeginExecute(new Uri(qryUri, UriKind.Relative), r>
{
try
{
var results = Ctx.EndExecute<Event>(r);
}
catch (Exception ex)
{
// TODO
}
}, null);
If you are an OData veteran, you might have gotten used to the LINQ syntax. I expect this to make a return, but in this early version it doesn't work and might not in the supported release that is coming soon. There are limitations on the platform (reflection emit is the big problem) that makes that much harder.
The last thing I had to deal with was that consuming this data was a challenge for a variety of reasons. The data that I have in the service was primarily designed for use on an HTML page therefore some of the data was formatted as HTML. This wouldn't be too tricky if Silverlight for the Windows Phone 7 contained the RichTextBox from Silverlight 4, but it doesn't so I opted for a simple converter that stripped the HTML out.
For the outline (which is really based on two related tables, not a HTML-based outline list) it was easier with nested DataTemplates:
I find that using I used the ItemsControl a lot more since it doesn't have user interface chrome around it but is simply an easy 'repeater'.
Since I was downloading all that data, I wanted to keep it around as state. Unfortunately the proxy created via the tools was using the DataServiceCollection class which doesn't play well with the serialization scheme using for tombstoning. (If you're not familiar with that phrase, see my blog entry on it here). I tried creating the OData proxy without it but there were other issues so I backtracked and edited the proxy by hand. I simply replaced all DataServiceCollection<T> references with List<T>. I also had to use the IgnoreDataMember attribute for some utility properties I added on the data objects:
public partial class Workshop
{
[IgnoreDataMember]
public string Link
{
get
{
return string.Concat("https://agilitrain.com/Workshop/Outline/",
Name.Replace(" ", "_"));
}
}
...
}
While this fixed the tombstone serialization issues, I found that when I used the WebBrowserTask (or perhaps other tasks too) that when returning to the application that it was locking up during re-hydrating the objects. I am still not sure why not but I've sent MS some information that I hope will help. Currently I chose to not save the data during tombstoning and just reload it from the web.
The last thing I want to mention is that when I refactored it to use the page navigation I had a good idea. I'd use a query string on the navigation so I could determine what information I needed in the details page. This is a pretty good idea and in fact there are several projects that are creating routing frameworks (like the one in ASP.NET and RoR) to help with this. But in this early pass, I decided to handle it myself. But the problem was that the Uri class doens't parse query strings (or other parts) of relative URI's so I had to hack it manually. Not hard, but annoying:
protected override void OnNavigatedTo(NavigationEventArgs e)
{
base.OnNavigatedTo(e);
...
// Hack together a parsing a query string.
var url = e.Uri.OriginalString;
var qry = url.Substring(url.IndexOf('?') + 1);
var pieces = qry.Split('=');
var id = int.Parse(pieces[1]);
...
}
This way I could retrieve the ID of the Workshop to display. I expect to use a routing framework as time goes on, but in this early stage I am ok with a little hacking to get it to work.
I hope that my story of building this app will help you see some of the changes you need to think about when designing your phone applications and you won't just simply try to port desktop/web apps to the phone. The promise is there to have a single codebase for all three but its not realistic to me. The difference in the touch interface, screensize and smaller device (e.g. CPU and memory) means that its an entirely different experience on the phone. You can re-use assets that you have (e.g. like my OData service) but the experience design is completely changed.
What do you think?
Shameless Plug:
If you are building phone apps and you need some help doing so, don't forget that I am teaching two Silverlight for the Windows Phone 7 Workshops coming up before the end of the year for my company AgiliTrain:
While I was giving my OData talk, someone asked about consuming OData on the WP7 phone. I had done this on the CTP earlier, but hadn't tried it during the beta. So I figured I'd look into it today. While this is still pretty easy to do, the tooling still isn't in place. This means that you can't simply do an "Add Service Reference" to a Windows Phone 7 project. Instead you have to follow these steps:
Unzip the Windows Phone 7 OData Library to get access to the reference.
In your project, add a reference to the System.Data.Services.Client.dll assembly from the .zip file.
Create the proxy classes by using the .NET 4.0's DataSvcUtil.exe (located in the %windir%\Microsoft.NET\Framework\v4.0.30319 directory). See the example below:
Include this new proxy class (containing the context object and the data classes) in your WP7 application.
Use the class to call out to the service as necessary:
NetflixCatalog ctx =
new NetflixCatalog(new Uri("http://odata.netflix.com/Catalog/"));
var qry = from p in ctx.People
.Expand("TitlesDirected")
where p.Name == "Stanley Kubrick"
select p;
// Create DataService query since we're not
// using a DataServiceCollection
var dsQry = (DataServiceQuery<Person>)qry;
dsQry.BeginExecute(r =>
{
try
{
var result = dsQry.EndExecute(r).FirstOrDefault();
if (result != null)
{
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
Titles = result.TitlesDirected;
});
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}, null);
Note that you need to use a Dispatcher as this call is *not* guaranteed to be called on the UI thread. This was the one piece that was unexpected. Once I marshaled the update to the UI thread, all worked perfectly.
UPDATE, Removed Embedded Slides for simply links as the Flash apps where killing my site perf.
I just got back from DevLink and had a great time. Great audiences as usual. I promised the attendees that I'd share my slides and code so here they are:
I've ported my XBoxGames Database (see this blog article for copies of the .mdf files) to SQL Azure and added OData support. You can find the feed here:
While this rocks. it did expose some of the lack of full OData support in SQL Azure but they're working on them immediately. Michael Pizzo is working on this support and send this list of what is and isn't working currently:
Supported:
Requests of the form “http://odata.sqlazurelabs.com/OData.svc/v0.1/[SQLAzureServerName]/[SQLAzureDatabaseName]” are recognized and correctly routed
Supports PowerPivot, Astoria Client/ASR, LinqPad, etc.
Integration with Portal, ACS
Impersonation based on security token for authenticated access
Anonymous access mapped to a particular user (read-only reference data, browser access,…)
Configuration information is read from a configuration table, hosted in sql azure
Metadata is dynamically loaded (changes to SqlAzure schema are immediately available through OData endpoint)
Pluralization support for entityset naming
IDs based on PKs
Relationships identified based on FKs
Goal is to generate an equivalent model to what EntityDesigner generates by default when doing modelgen from a database
The following requests are supported:
ServiceDocument (/)
metadata (/$metadata)
entityset (/entitysetname)
identity operator (/entitysetname(id))
property navigation (/entityset(id)/propertyname)
count (/$count)
single segment relationship navigation (/entityset(id)/navigationpropertyname)
The following query operators are supported:
$top
$skip
$orderby
$filter with basic expression support
$inlinecount
$select
$skiptoken
ServerDriven paging is supported
first 50 records plus NextLink
The following are supported in expressions:
math operators (add, subtract, multiply, divide, mod)
A host of string functions (length, substring, left, right, trim, replace, remove, upper, lower, startswith, endswith, concat, indexof)
Basic updating support
user-defined key values
autoincrement key values
Not supported (yet): Stored procs Tables/views w/no primary key Date and Math scalar functions EntityClient support $expand Many:Many relationships Multi-level relationship navigation Media Link Entries Other feature requests?
Other Bugs:
Not listed in Michael's notes are a couple of other issues i've run into (but i've reported them to him so I expect they're going to be addressed soon):
I'd recently been asked by Chris Sells to help him with a simple WCF Data Services/jQuery example so I thought I'd share it via my blog as well. The basic idea is to use jQuery's AJAX functionality to retrieve JSON instead of the usual OData and consume it on a web page.
The example I decided on using is to expose the XBox database with paging. I am not doing any of the niceties like getting result counts to show a real navigation bar. Instead this is quick and dirty to simply do "next" and "prev" and use WCF Data Service's URI API to retrieve data.
Like most of my XBox examples, I am using the Entity Framework to simply expose three entities types (Games, Genres and Ratings). Then I utilize a WCF Data Service to expose these types for consumption via REST. My HTML is simple:
The gameTable will simply be filled with a simple TABLE element for our data. So to the jQuery we go. I am using plain jQuery 1.3.2 (though plugins could be used to simplify some of this code, I choose to just do it raw for simplicity).
First I set up a couple of variables to hold our paging information:
// globals for paging
var page = 0;
var pagesize = 25;
Next I handle the document's ready function to do some work when the initial page load is complete:
// Loads once the document has completed loading
$(document).ready(function () {
// Set up paging buttons
$('#PrevPage').click(function (evt) {
// Stop the navigation from happening
evt.preventDefault();
if (page > 0) {
page--;
loadData();
}
});
$('#NextPage').click(function (evt) {
evt.preventDefault();
page++;
loadData();
});
// Load the initial data
loadData();
});
When setting up the paging buttins, I use the '#PrevPage' and '#NextPage' CSS selectors to find items named PrevPage and NextPage and wire up the event to cause the page changes to happen. The 'evt.preventDefault()' call stops the buttons from trying to navigate since we're using them as buttons. FInally once we change the current page value, we call loadData (or in the initial case, call loadData with the default values).
In the loadData function, I first create the URL by using the Data Service/OData URL API to retrieve games by Name while using the global values to retrieve a page of data at a time (see the $skip and $top parameters). Once the URL is defined, jQuery's ajax function executes the URL and returns in the success function in which we call loadTable to create the table for us. Since the returned JSON is contained in a 'd' element for security, we de-reference the 'd' element before we sent it into the loadTable function.
function loadTable(results) {
var table = '<table><thead><tr><th>Name</th>' +
'<th>Publisher</th><th>Box</th></thead><tbody>';
for (var post in results) {
var row = '<tr>';
row += '<td>' + results[post].Name + '</td>';
row += '<td>' + results[post].Publisher + '</a></td>';
row += '<td><img src="' + results[post].ImageUrl +
'" style="width: 100px; height=150px" /></td>';
row += '</tr>';
table += row;
}
table += '</tbody></table>';
$('#gameTable').html(table);
}
Finally in loadTable some simple HTML construction goes on to create a table with our results. The for loop returns a iterator for each row in the collection which we can use to retrieve the individual item (see results[post].Name as an example). Once the table is built, it replaces the contents of the element named gameTable with the new table.
Because WCF Data Services will return JSON as well, you can use it to do AJAX work just as effectively as Silverlight can consume it for rich client work. You can get the code here:
