The version redirection in BizTalk do NOT work as one is used to coming from a ordinary .NET developer background (so I was wrong in this post ...). Say that we made a reference to a code library in one of our orchestrations. This code library is in version 1.0.0.0 when we build and deploy the orchestration to BizTalk and to the GAC. The setup in VS 2005 looks something like this.

Then we make some minor changes in the code library, we fix them and set the version to 1.0.1.0. Build and deploy it. This means that we now have one 1.0.0.0 and one 1.0.1.0 version side-by-side in the GAC (as shown in the figure below).

Coming from .NET the CLR should now load the 1.0.1.0 version as it identifies assemblies based on the assembly name, major or minor version (NOT the build and revision version), public key token and culture (more about this here).

However, this does NOT work in BizTalk. BizTalk loads assemblies by fullname it has stored in the management database and the build and revision number are part of the fullname ... This means we have to build and redelopy EVERY part where we like to use new version of the code library.

There is one way around this for emergency use. Say that one has a code library that is used in loads of BizTalk artifacts (A place where we store all base functionality for the solution). Rebuilding all those parts and redeploying it, just to be able to update the version number, is not going to happened! It's to much work and to risky. So it's possible to make a change in the BTSBTSvc.exe.config file instead. In my case the change would be something like the below (read more about here).

<div><span style="color: #000000; "><</span><span style="color: #000000; ">dependentAssembly</span><span style="color: #000000; ">></span><span style="color: #000000; ">
  </span><span style="color: #000000; "><</span><span style="color: #000000; ">assemblyIdentity name</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">StaticCodeLibrary</span><span style="color: #000000; ">"</span><span style="color: #000000; "> publicKeyToken</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">32ab4ba45e0a69a1</span><span style="color: #000000; ">"</span><span style="color: #000000; "> culture</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">neutral</span><span style="color: #000000; ">"</span><span style="color: #000000; "> </span><span style="color: #000000; ">/></span><span style="color: #000000; ">
  </span><span style="color: #000000; "><</span><span style="color: #000000; ">bindingRedirect oldVersion</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">1.0.0.0</span><span style="color: #000000; ">"</span><span style="color: #000000; "> newVersion</span><span style="color: #000000; ">=</span><span style="color: #000000; ">"</span><span style="color: #000000; ">1.0.1.0</span><span style="color: #000000; ">"</span><span style="color: #000000; ">/></span><span style="color: #000000; ">
</span><span style="color: #000000; "></</span><span style="color: #000000; ">dependentAssembly</span><span style="color: #000000; ">></span></div>

As I don't use regular expressions that often I always forget the syntax. So I thought I just put a basic replace pattern up here.

This method takes a schema, finds all places where is says schemaLocation="whatever" in a text and changes this to schemaLocation="whatever.xsd" and then returns the schema.

<div><span style="color: #0000FF; ">private</span><span style="color: #000000; "> XmlSchema FixSchemaLocation(XmlSchema schema)
{
  System.Text.RegularExpressions.Regex locationReplacePattern </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> System.Text.RegularExpressions.Regex(</span><span style="color: #000000; ">"</span><span style="color: #000000; ">schemaLocation=\"(?<location>.*?)\"</span><span style="color: #000000; ">"</span><span style="color: #000000; ">);
  </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> locationReplaceValue </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #000000; ">"</span><span style="color: #000000; ">schemaLocation=\"${location}.xsd\"</span><span style="color: #000000; ">"</span><span style="color: #000000; ">;
  </span><span style="color: #008000; ">//</span><span style="color: #008000; ">Puts .xsd after the schemaLocation. We need this find the imported schemas</span><span style="color: #008000; ">
</span><span style="color: #000000; ">  StringWriter sw </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> StringWriter();
  schema.Write(sw);
  XmlSchema formatedSchema </span><span style="color: #000000; ">=</span><span style="color: #000000; "> XmlSchema.Read(</span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> StringReader(locationReplacePattern.Replace(sw.ToString(), locationReplaceValue)),</span><span style="color: #0000FF; ">null</span><span style="color: #000000; ">);
  </span><span style="color: #0000FF; ">return</span><span style="color: #000000; "> formatedSchema;
}</span></div>

I'll start trying to post a weekly sum up of BizTalk, .NET topics I've come across during the week. We'll see how goes as I'm not the most constant blogger ...

However I worked with mainly three things this week.

Error handling and General Exception

I've tested some ideas relating to an error handling pattern we like to implement in a one of our major integration projects. We've looked at how the new BizTalk 2006 Failed Message Routing fits in with our project. The idea is to have and Orchestration listening to the Error Message, log these and then put the failing message parts to file. We'll then have a system that picks up the failed messages, launches them in an editor and makes it possible to resend the message after it's been edited.

Problem arises when we get to handling exceptions in Orchestrations. Orchestrations doesn't have the possible to use Failed Message Routing as ports does (I guess that ok, we might want handle different errors in different ways and so on.). However that means that we have to catch an error and create our own Error Message to submit to the message data box for further routing to the Orchestration handling errors (the one that logs and puts the message to file.) So far so good as long as we don't get an General Exception! Usually when getting an General Exception, and we don't catch it, the message gets suspended and we get a some error details giving us an idea of what went wrong. However as soon as one catches a General Exception we loose all information about the error ... We've still haven't found a good way of handling these errors. We just can't afford loosing that error information!

General Exceptions are still a mystery to me. I guess that are there so BizTalk has a chance in catching exception that aren't raised from .NET based code. But what kind of errors within an orchestration raises these kind of exceptions? I know a failure in a mapping does, what else? I'd really like this scenario to better documented. The error handling patter is based on a chapter in the Pro BizTalk 2006 book (which is a excellent book in my opinion - buy it!) but the General Exception and how to deal with it isn't discussed there either. Any ideas, book, articles are highly appreciated!

Validation

One important thing to think of in a BizTalk solution is to not let anything either in or out that doesn't validate (ok, there are exception to this, but generally speaking)! BizTalk has great built in support for this in the XML Validator Pipeline Component (you might also want to have a look at this code). However when working with this it's important to understand what one is validating against and it suddenly even more important to understand every details in the schemas (external system owners usually like some technical explanation when your telling them that their messages don't validate in your pipeline ;)). Things we run into this week is the elementFormDefault attribute. The following reading help me understanding what the attribute does:

• http://www.w3.org/TR/xmlschema-0/#NS
• http://www.hanselman.com/blog/XmlValidating...XmlReaders.aspx
• http://geekswithblogs.net/dmillard...12935.aspx
• http://blogs.msdn.com/ebattalio/archive/2006/03/03/543154.aspx

Another thing we ran into was white space handling and the xml:space attribute. Apparently if one likes to have a node with a space in it BizTalk removes this if the element doesn't have the xml:space attribute. so will come out as - sound familiar?

But if the schema doesn't declare that the node will have a xml:space attribute the validation will fail! To get this working one has to get the schema declaring the the xml:space attribute and then reference it like this example. The schema from w3c (if you don't write your own like in the article) is located here.

End-to-end tracing in a SOA

I'll make this short even if it should be the longest part in this post. Basically we're trying to archive the following:

Our solution send loads of different messages types between five different BizTalk servers. The client likes to be able to have full text search within these messages and to also be able to see all the messages in each interchange within a server.

So say for example that we receive and Order message in a flat file format. This is transformed in to a Xml message that is then routed down to two other different BizTalk servers. It should then be possible for the client to start an application, click on the Order message type, enter for example the order number (an element i in the message) and see the the full content of the different files within the interchange (in this example that would be the flat file and the Xml file) where that Order number is found.

Basically it's does what the HAT does (with full text search and a custom GUI). In the first phase of the project we'll have it work per server, but in the final solution this should perform over all servers ... I'll come back to this in a later post. In the mean time listen to this episode of Hanselminutes on end-to-end tracking. This will be a challenge in every service orientated architecture ...

Finally

I've sold my iPod Nano and ordered the Creative Vision Zen:M. Now I'll catch up on all those dnrTV Webcasts! I'll also make sure to watch everyone of the BizTalk 2006 Webcasts during my commute to work.

Let me know if you found some of this information useful and I'll try harder to post some like this every Friday.

Convoying is one of BizTalk's real strengths but it has a lot of pitfalls (I'm talking performance, zombies etc). This article is really good at explaining the different patterns used for creating convoys. It also makes some deep dives into how the subscriptions are solved, why zombies are created and how to deal with them.

Even if you feel you understand the convoy patters from before the part about how BizTalk solves the subscriptions for correlations (the part called Basic Convoy Theory) is great.

Using sequential convoy to handle ordered delivery

A big selling point in BizTalk 2006 is it's ability to handle Ordered Delivery. It's important however to understand that this setting (on the Receive and Send port) only works in a pure messaging scenario (a scenario without orchestration, just passing messages between ports). To get order delivery in a scenario using orchestrations one has to use the sequential convoy pattern (basically forcing the orchestration to only one instance on one thread). This Webcast explains this in depth and also deals with some of the different problems that are related to the issue. Problems like performance of course, but also the requirement for receive adapter (on the send side all adapters support ordered delivery) for enabling ordered delivery (you'll have to use MSMQ, MQSeries or specific scenarios of File, SOAP or HTTP).

Both convoying and ordered delivery are important concepts to understand in depth to be able to make the right decisions in a BizTalk 2006 solution.

Recently I had to insert a record using a stored procedure and the SQL Adapter in BizTalk 2006. There are lots of examples on both how to insert records and how to select a number of record using this adapter. However I had problems finding how to insert a record and receiving the new id of the inserted row in return (the SCOPE_IDENTITY()). In my scenario I needed the id to insert into another database further down in the orchestration.

I ended up with a stored procedure looking like the below.

<div><span style="color: #0000FF; ">ALTER</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">PROCEDURE</span><span style="color: #000000; "> </span><span style="color: #FF0000; ">[</span><span style="color: #FF0000; ">dbo</span><span style="color: #FF0000; ">]</span><span style="color: #000000; ">.</span><span style="color: #FF0000; ">[</span><span style="color: #FF0000; ">TestInsertParty</span><span style="color: #FF0000; ">]</span><span style="color: #000000; ">
    </span><span style="color: #008000; ">@partyName</span><span style="color: #000000; "> </span><span style="color: #000000; font-weight: bold; ">nchar</span><span style="color: #000000; ">(</span><span style="color: #800000; font-weight: bold; ">30</span><span style="color: #000000; ">) </span><span style="color: #808080; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">NULL</span><span style="color: #000000; ">
</span><span style="color: #0000FF; ">AS</span><span style="color: #000000; ">
</span><span style="color: #0000FF; ">BEGIN</span><span style="color: #000000; ">
    </span><span style="color: #0000FF; ">SET</span><span style="color: #000000; "> NOCOUNT </span><span style="color: #0000FF; ">ON</span><span style="color: #000000; ">;
    </span><span style="color: #0000FF; ">Insert</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">Into</span><span style="color: #000000; "> Party (</span><span style="color: #FF0000; ">[</span><span style="color: #FF0000; ">name</span><span style="color: #FF0000; ">]</span><span style="color: #000000; ">, chain_idx) </span><span style="color: #0000FF; ">Values</span><span style="color: #000000; ">(</span><span style="color: #008000; ">@partyName</span><span style="color: #000000; ">, </span><span style="color: #0000FF; ">NULL</span><span style="color: #000000; ">)
    </span><span style="color: #0000FF; ">Select</span><span style="color: #000000; "> </span><span style="color: #FF00FF; ">Scope_Identity</span><span style="color: #000000; ">() </span><span style="color: #0000FF; ">As</span><span style="color: #000000; "> Id </span><span style="color: #0000FF; ">For</span><span style="color: #000000; "> Xml </span><span style="color: #000000; font-weight: bold; ">Raw</span><span style="color: #000000; "> (</span><span style="color: #FF0000; ">'</span><span style="color: #FF0000; ">Response</span><span style="color: #FF0000; ">'</span><span style="color: #000000; ">)
</span><span style="color: #0000FF; ">END</span></div>

The trick was to use the XML RAW Mode. This mode transforms the result set into a generic identifier as . It is however possible to provide a element name, as <_Response>_. Basically this will insert the new value and return something like this from the stored procedure.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">Response </span><span style="color: #FF0000; ">Id</span><span style="color: #0000FF; ">="1054"</span><span style="color: #FF0000; "> </span><span style="color: #0000FF; ">/></span></div>

After return via the send port the orchestration will receive something like the below.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">TestInsertResponse </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">="TestInsert"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Response </span><span style="color: #FF0000; ">Id</span><span style="color: #0000FF; ">="1054"</span><span style="color: #FF0000; "> </span><span style="color: #0000FF; ">/></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">TestInsertResponse</span><span style="color: #0000FF; ">></span></div>

The schema that I use to both handling the response and request against the SQL Adapter is shown below. First I set the type of the Id-attribute to xs:int but this gave me some problems when using the promoted value in the orchestration, everything worked fine when switching back to xs:string.

The same technique would be used for receiving a code from the stored procedure (say 1 for success and 0 for failure or whatever) and then to make a logical decision in the orchestration.

I've just finished reading a paper written by David Pallmann.pdf) (PDF) from Neudesic. Belive it or not (or even better check it out), it's a short and understandable description of what an ESB is!

Neudesic sells a product called Cyclone which is something they call a ESB architecture and framework software and even if describing this product is the main purpose of the paper the first part deals with describing the basics of an ESB and try's to "assemble a synthesis of popular ESB definitions".

I've chosen a couple of parts of the paper that really appealed to me but I really recommend the full length version (only 8 pages!).

Defining the ESB >> * An ESB is a backbone for connecting and integrating an enterprise’s applications and services. > * An ESB provides the necessary infrastructure to create a service oriented architecture. > * An ESB is a convergence of EAI, MOM, and SOA concepts. > * An ESB is based on open standards such as XML, SOAP, and WS-*. > * An ESB provides intelligent routing, such as publish-subscribe, message brokering, and failover routing. > * An ESB provides mediation, overcoming data, communication, and security differences between endpoints. > * An ESB integrates with legacy systems using standardsbased adapters. > * An ESB provides logical centralized management but is physically decentralized. > * An ESB is able to apply EAI concepts such as rules and orchestrations. > * An ESB is able to monitor and throttle activity as per a Service Level Agreement (SLA). >

I never thought of an ESB as an convergence of EAI (Enterprise Application Integration), MOM (Message Oriented Middleware) and SOA (Service Oriented Architecture) concepts. But, ok, so what does the writer really mean by these concepts?

SOA makes loosely-coupled, decentralized solutions possible that are enterprise-ready and based on interoperable standards. EAI allows integration of any combination of systems, with sophisticated message brokering, message translation, business process orchestration, and rules engine processing. MOM provides intelligent routing such as publish-subscribe topical messaging and strong managerial controls over routing, auditing, activity monitoring, and throttling. >>

And why do we need a bus architecture on top of SOA, MOM and EAI?

Point-to-point architecture works all right on a small scale, but its problems become apparent when used at the enterprise level. If each system has to know the connection details of every other system, then each new system added increases the problem of configuration and management exponentially. This was the impetus that led us to hub-and-spoke architectures, which most EAI products use. This architecture was a vast improvement over point-to-point architectures, and each system needed to communicate with only the hub. In addition, the hub could provide excellent management features since it was a party to all communication. It only took time to reveal some shortcomings with the hub-and-spoke approach, and today it is often associated with concerns about scalability, single point of failure, and vendor lock-in. >> ... >> Fortunately, there is a sound compromise to be found in the bus architecture, which provides the benefits of logical centralization but is physically decentralized. The bus architecture in earlier days was often used in message bus systems based on proprietary technologies, but an ESB implements this architecture using WS-* standards. >>

So using all three of them will make an ESB? Well, not really ...

If combining disciplines was all there was to an ESB, we’d simply call it “consulting”. To properly leverage these disciplines they need to be combined in the right way through an architecture that lets their strengths shine and overcome their inherent weaknesses. Each discipline has some weak areas that the others help to resolve: SOA needs better enterprise manageability; EAI needs to become decentralized; MOM needs to get away from proprietary technologies. Combining these disciplines properly in an ESB overcomes these weaknesses. >>

I'd be very intreseted in comments of other artciles, papers and so on relating to the ESB concept!

Ever had a "use of unconstructed message" error message in your orchestrations? Then you know that messages has to be constructed before one can use them within orchestrations. Basically messages in orchestrations are created in ports or when executing a map in a construct shape. However there is also the possibility of creating messages in expression shapes using code. Matt Meleski does a great job explain the different approaches in his Constructing BizTalk 2004 XML Messages (In an Orchestration) - Choices blog post.

However there are some pitfalls when constructing messages from scratch within orchestration (a part from the fact that there really isn't a clean way of doing it ... Scott Colestock has some ideas in this post).

A common problem is XPathUpdateException. Say one has a mapping that looks like the one below where one node isn't mapped. This will create a message without the ElementB element. This doesn't have to be in a mapping, it could also happend when constructing a message using code in a expression shape. A common scenario is that we add code for constructing the bone structure of the XML message, then change the schema without updating the construct code. Ergo a message will be constructed without that element (could be both valid or invalid XML then depending on the Min Occurs value)!

[](../assets/2007/05/windowslivewriterxpa.setdistinguishedfieldwhileassigningv-ceb2complexmapping23.jpg)

When we then try to add a value to the element that hasn't been created the exception will be thrown!

One way around when using a map is to make sure to have a default value on every element to force them be constructed. Adding the following to map will make it work.

It's a one time mistake but it's good to know what to look for when the XPathUpdateException in Microsoft.XLANGs.Core.XSDPart.SetDistinguishedField (if it's a distinguished field that is) shows up in the event log.

Ever seen this alert when trying to debug a project using Visual Studio 2005?

The following module was build with optimizations enabled or without information: >> C:\Windows\assembly\GAC_MSIL\ ... >> To debug this module, change its build configuration to debug mode. TO suppress this message, disable the 'Warn if no user code on launch' debugger option. >>

Basically this means that Visual Studio can't find the debug file (the .pdb file) at the same location as the dll is loaded from. Reading the error message tells us that the dll been loaded from the GAC and there aren't any pdb files there! Just make sure to uninstall the dll from the GAC and Visual Studio should be able to load it from your Debug folder and you'll probably be good to go.

Update 2007-07-05: The example project used in the post can be downloaded from here.

The operations dll (Micrsoft.BizTalk.Operations) is one of the new bits in BizTalk 2006 that at least I haven't heard much about (most of you probably find in the C:\Program Files\Microsoft BizTalk Server 2006 folder). However it's a library with a lot of useful functionality.

In this post I'll focus on how it's possible to use the library to get hold of message parts and message context from the BizTalk tracking database (usually called the BizTalkDTADb). If you're unfamiliar with the architecture of the tracking in BizTalk this article is a good place to start.

What used to be the problem?

In BizTalk 2004 one soon got into problems when trying to read the message parts and the message context from the tracking database. The problem is that the the Xml is compressed (something that makes totally sence - Xml is a perfect candidate for compression). To my knowledge no one has found a good way to decompress the message context. There's however a way to decompress the message parts (and only the parts) that Rob posted on in the BizTalk newsgroup. But this method doesn't work on the compressed context of the same message that the parts belong to! One could assume that the same method could be used for both compression and decompression but I haven't got it to work for the context of the message (read about my frustration here)

What's wrong with WMI and MSBTS_TrackedMessageInstance?

So the option that was left to us before BizTalk 2006 for reading all the message parts as well as the message context was to use the MSBTS_TrackedMessageInstance WMI scrtipt and save everything down to file. The problem with this approach is that it's slow and ugly! Say that we like to save a couple of thousand messages from the tracking database. When forced to save the messages to file we'll end up with a slow and ugly solution where writing to file takes ages, then we have to read the content of the different files before cleaning up and deleting the files.

Micrsoft.BizTalk.Operations to the rescue

A quick view in Lutz Roeder's Reflector shows us a couple of interesting methods in the operations dll. However I'll only use the GetTrackedMessage method in this post.

I've written a tiny test application that uses the library to read the body part of the message (the meat of the message) and a couple of properties I'm interested in from the context of the message. It looks something like this. If your interested drop me an email and I'll send it. The first screenshot below shows an example of reading the InterchangeID from the context of a tracked message.

This screenshot shows an example of reading the full body of a tracked message from the tracking database. Try doing this with less then 10 lines of code using the MSBTS_TrackedMessageInstance script!

There is really nothing to the application, reading a tracked message using the MessageID will return an object implementing IBaseMessage and we basically read the BodyPart property of - it really couldn't be any easier.

<div><span style="color: #0000FF; ">public</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">static</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> GetMessageBodyByMessageID(</span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> dbServer, </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> dbName, Guid messageID)
{
    TrackingDatabase dta </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> TrackingDatabase(dbServer, dbName);
    BizTalkOperations operations </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> BizTalkOperations();
    IBaseMessage message </span><span style="color: #000000; ">=</span><span style="color: #000000; "> operations.GetTrackedMessage(messageID, dta);
    </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> body </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">string</span><span style="color: #000000; ">.Empty;
    </span><span style="color: #0000FF; ">using</span><span style="color: #000000; ">(StreamReader streamReader </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> StreamReader(message.BodyPart.Data))
    {
        body </span><span style="color: #000000; ">=</span><span style="color: #000000; "> streamReader.ReadToEnd();
    }

    </span><span style="color: #0000FF; ">return</span><span style="color: #000000; "> body;
} </span></div>

I'd be very interested in how people use the operations library both when it comes to read and use tracked messages, but also in other ways! Use the comments to tell me and other readers how you use it your solutions!

I've just found out that I get to go to TechEd in November! I haven't been there before. I've been to a lot of conferences but not the Microsoft conference. Do I have to say that I'm looking forward to it?

If you're going and feel like meeting up for a BizTalk lunch, dinner or whatever don't hesitate to drop me an e-mail.

>> **Update 2012-03-28** Unfortunately the images for this post was lost in upgrade of the blog. >>

This post shows how it's possible to create your own .NET class and then use this within your orchestration (as a variable) to work with typed lists - something that unfortunately isn't supported out of the box in BizTalk orchestrations. Below is the class used to create a typed list for System.String objects.

<span class="kwrd">using</span> System;
<span class="kwrd">using</span> System.Collections.Generic;
<span class="kwrd">using</span> System.Text;

<span class="kwrd">namespace</span> Sample.BizTalkCollections
{
    [Serializable]
    <span class="kwrd">public</span> <span class="kwrd">class</span> StringList
    {
        <span class="kwrd">private</span> List<<span class="kwrd">string</span>> _list = <span class="kwrd">new</span> List<<span class="kwrd">string</span>>();

        <span class="kwrd">public</span> <span class="kwrd">void</span> Add(<span class="kwrd">string</span> item)
        {
            _list.Add(item);
        }

        <span class="kwrd">public</span> <span class="kwrd">int</span> Count()
        {
            <span class="kwrd">return</span> _list.Count;
        }

        <span class="kwrd">public</span> <span class="kwrd">bool</span> Remove(<span class="kwrd">string</span> item)
        {
            <span class="kwrd">return</span> _list.Remove(item);
        }

        <span class="kwrd">public</span> <span class="kwrd">void</span> RemoveAt(<span class="kwrd">int</span> index)
        {
            _list.RemoveAt(index);
        }

        <span class="kwrd">public</span> <span class="kwrd">override</span> <span class="kwrd">string</span> ToString()
        {
            StringBuilder builder = <span class="kwrd">new</span> StringBuilder();
            <span class="kwrd">foreach</span> (<span class="kwrd">string</span> item <span class="kwrd">in</span> _list)
            {
                builder.AppendLine(item);
            }

            <span class="kwrd">return</span> builder.ToString();
        }
    }
}

Notice that the class is serializable so BizTalk can serialize when dehydrating the orchestration to the database. We've also chosen not to extend the IList interface as we don't want to expose all methods of that interface, but only the once we'll really going use within our orchestrations.

This StringList class exists within the BizTalkCollections namespace where we have other classes for lists types with different datatypes (for example ObjectList, Int32List and so on). Below is a picture showing the structure of the Visual Studio project containing all the collections we current have implemented. This project and the BizTalk testproject is available for download here.

In the orchestration we then have set up a variable and type this to our StringList class within the Sample.BizTalkCollections namespace.

Also make sure the assembly with your collection class is installed in the GAC, otherwise you'll end up with a FileNotFound exception saying

>> Could not load file or assembly 'Sample.BizTalkCollections, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7109528d5dbf986b' or one of its dependencies. The system cannot find the file specified. >>

Then you can use the following code within your orchestration! Download the sample project and test it.

A really generic class

I'd really like for it be possible to create my own generic classes and then initialize these using something like the below.

genericList = Sample.BizTalkCollections.GenericList<System.String>();

However this isn't possible as BizTalk orchestrations don't seem to understand generic classes ... If someone found a solution to this please let me/us know via the comments to this post.

The XPath function that's available directly inside BizTalk orchestration is a powerful little tool. However I've seen a couple of project where developers just grown tired of it and started creating their own little libraries instead. I'll be the first to admit that the XPath function isn't perfect, and it sure doesn't work like most of the other XPath engines (which is the biggest problem) but it's still inside the orchestration and you can use it to both read and assign values to a message which is super useful! Basically I don't see a valid reason for bringing more complexity into your solution by adding another library - as long as you're just going to read or set value using XPath.

However there is one trick that you should know of when it comes to reading a text value from a node. Basically you have to use both the string() and
text() XPath functions. Both Charles Young and Yossi Dahan has good post on this subject. Also if your new to writing XPath expressions for complex schemas with loads of namespaces and stuff (like schemas in BizTalk) this post could be useful for you.

Finally a nice tool for writing and testing small XPath expression inside Visual Studio (if you don't want to spend x minutes waiting for XmlSpy to start up ...) is XPathmania. Read about it here - I use it all the time!

The main problem I have with developing BizTalk orchestrations is the fact that I'm so blind when it comes to follow the runtime processing. Using the debugger that is part of the HAT tool is slow and clumsy which IMHO makes the tool almost useless in everyday development. But there is hope!

DebugView

Sysinternals (Windows Sysinternal now - Microsoft bought them last year) DebugView is a wonderful little tool and is especially useful when it comes to figure out what's actually going on inside an orchestration. Basically the tool listens to system wide debug output. From an orchestration it's possible to write debug information using the .NET System.Diagnostics namespace and the Debug or Trace class.

Decide on how to filter

There are a couple of handy little tricks that makes DebugView a even better in BizTalk development. First one should try and have something in the debug messages that makes it possible to filter and distinct one's own (as DebugView listens system wide debug output all running applications debug info will show up). Our team decided on "Sogeti" (our company name) for all our development and to have a method in our BaseLibrary component that outputs something like the below (the BaseLibrary is a small little .NET component with a couple of very useful classes we use company wide in our BizTalk related development).

<div><span style="color: #000000; ">System.Diagnostics.Debug.WrtieLine(</span><span style="color: #000000; ">"</span><span style="color: #000000; ">Sogeti, your debug/trace message here</span><span style="color: #000000; ">"</span><span style="color: #000000; ">)</span></div>

This make is possible to have a filter in DebugView and to for example have it look something like this.

Trace full context of messages

Another little useful trick is to trace the full context of messages. This is done be storing the message in a XmlDocument typed variable and get the OuterXml property of that variable. The below code is and example of this.

<div><span style="color: #000000; ">tempXml </span><span style="color: #000000; ">=</span><span style="color: #000000; "> msgFindPartyRequest.parameters;
System.Diagnostics.Trace.Write(System.String.Concat(</span><span style="color: #000000; ">"</span><span style="color: #000000; ">Sogeti, msgFindPartyRequest: </span><span style="color: #000000; ">"</span><span style="color: #000000; ">, tempXml.OuterXml));</span></div>

Example of a full message trace.

To Trace or Debug - that's the question

As stated earlier both System.Diagnostics.Debug and System.Diagnostics.Trace has methods (Write, WriteLine and so on) for outputting debug information. However there is only one that stays in your compiled code when switching from Development to Deployment compilation mode (guess which one ;)). So make sure you choose the right class for the right information. I like to have some critical messages left using Trace and be able to trace these even on the test and production server.

DebugView on a remote desktop

When running DebugView on an other server (say a test or a production server) using Remote Desktop I've found that ones has to use the console user on the server. This kind of makes sense as if we're connection "normally" we're creating a virtual session and that's not were the debug information is written to.

Microsoft SOAP Toolkit 3.0

This is a totally other tool than DebugView but I thought it fit here any way. It's a handy tool when working with SOAP based messages. Without it's very hard to actually figure out how the raw request and response message look and why your orchestration web service is acting the way it does.

The trace tool is placed as a reverse proxy between BizTalk and the Internet. It's setup by telling the tracing tool which localhost port to listen at (for example 9091 as in the example below) then we'll redirect to that port by changing the setting in the BizTalk send port.

Finally we'll set up the trace tool to listen to port 9091 and redirect all traffic to our web service URL at port 80 in this case. So basically the trace tool will catch all the traffic hitting the 9091 port and forward it.

That's it! This is probably basic stuff for most of you but hopefully it's useful for someone!

I've also noticed that the SOAP Toolkit is deprecated by Microsoft and I'd like to hear if anyone used something else (like Fiddler example) for tracing SOAP messages. I'd also love some other tips, tools and methods you use for debugging BizTalk orchestrations.

There was a question on the Microsoft newsgroup the other day where someone had to split a message into parts. But some of the information that was supposed to go in to the different parts were part of the envelope. I thought I'd give my solution to the problem a try - here it is.

Say for example that we receive the following message.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">ns0:EmployeeSalesReport </span><span style="color: #FF0000; ">ReportID</span><span style="color: #0000FF; ">="R100"</span><span style="color: #FF0000; "> EmployeeID</span><span style="color: #0000FF; ">="0012345"</span><span style="color: #FF0000; "> xmlns:ns0</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.EmployeeSalesReport"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sales</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ns1:Sale </span><span style="color: #FF0000; ">xmlns:ns1</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">10</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ns1:Sale</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ns1:Sale </span><span style="color: #FF0000; ">xmlns:ns1</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">200</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">20</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ns1:Sale</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sales</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ns0:EmployeeSalesReport</span><span style="color: #0000FF; ">></span></div>

In the above message we have different sales information from one employee but all the global employee and report information (the ReportID and EmployeeID attributes) exists in the root node (what we'll call the envelope of the message).

What we like to achieve is to split this envelope message into it's different sale item so we get separate message looking something like the below where each item contains the global information from the envelope.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sale </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">200</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">20</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">EmployeeID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">0012345</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">EmployeeID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ReportID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">R100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ReportID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sale</span><span style="color: #0000FF; ">></span></div>

There might be different needs for doing this in a pipeline but much of it comes down to choice of architecture style in your BizTalk solution. Personally I'd refuse to introduce another orchestration for splitting a message like this. I don't believe that's what orchestrations are for (they should deal with possible logic and workflow in my world). I'm sure other people feel different - feel free to use the comments.

Step 1 - Transforming the envelope message using XSLT

The first thing we'll need to do is to transform the envelope message so that the information from the root node (the ReportID and the EmployeeID) get into every single Sale-node. There are a couple of ways of achieving this but I'll use the XSLT-transformation pipeline component that ships with the BizTalk 2006 SDK.

All this component does is to let you point out a XSLT stylesheet you like to use to transform your message.

The stylesheet I've used is part of the solution that you can download here. I will not show the stylesheet full script here but rather the result after the message travel through the component.

<div><span style="color: #0000FF; "><?</span><span style="color: #FF00FF; ">xml version="1.0" encoding="utf-8"</span><span style="color: #0000FF; ">?></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "><</span><span style="color: #800000; ">EmployeeSalesReport </span><span style="color: #FF0000; ">EmployeeID</span><span style="color: #0000FF; ">="0012345"</span><span style="color: #FF0000; "> ReportID</span><span style="color: #0000FF; ">="R100"</span><span style="color: #FF0000; "> xmlns</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.EmployeeSalesReport"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sales </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sale </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">10</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">EmployeeID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">0012345</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">EmployeeID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ReportID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">R100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ReportID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sale</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Sale </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">="http://Example.TransformAndSplit.Sale"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ItemID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">200</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ItemID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">Amount </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">20</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Amount</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">EmployeeID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">0012345</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">EmployeeID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
            </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">ReportID </span><span style="color: #FF0000; ">xmlns</span><span style="color: #0000FF; ">=""</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">R100</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ReportID</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
        </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sale</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
    </span><span style="color: #0000FF; "></</span><span style="color: #800000; ">Sales</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">EmployeeSalesReport</span><span style="color: #0000FF; ">></span></div>

Ok, nice. Now we have all the information we like in each of the Sale-nodes! All we have to do now is to split the message!

Step 2 - Splitting the message

We'll use the standard XmlDisassembler component for splitting the message. All we have to do is to make sure we done the following.

1. Set the schema envelope property to "true" in the schema editor.

2. Set the Body Xpath property on the EmployeeSalesReport-node to point to the Sale-node.

3. Configure the Document Schemas and Envelope Schemas (same properties are called DocumentSpecNames and EnvelopeSpecNames if your doing the configuration after deployment in the administration console) properties of the XmlDisassember component to match the names of your schemas.

This is what the envelope schema looks like in this example.

This is what the XmlDisassembler configuration looks like.

There, we're done! Now we can drop a test file containing two Sale-nodes and all the report/employee information on the top and have the two following separate files as a result when using a file send port.

Setting up the test solution

1. Download a zipped version of the solution.

2. You'll have to unzip the project to C:\Example.TransformAndSplit. If that isn't possible you'll have to change the path to the XSLT stylesheet in the XsltComponent in the pipeline as this is a fixed path.

3. When you build the XsltComponent the Output path is set to the Pipeline Components folder. This assumes that BizTalk is installed at _C:\Program Files\Microsoft BizTalk Server 2006_. If that isn't the case make sure to change the output path of the component.

4. If you'd like to run the pipeline component in debug I've set this up using the pipeline.exe tool. There are however some paths in the debug setting of the XsltComponent project also that assumes that your BizTalk solutions is installed at C:\Program Files\Microsoft BizTalk Server 2006. If this isn't the case you'll have to change some values in the Command lines arguments property, but that only if you like to run debug.

5. Build and deploy.

6. Set up a receive port and location using the TransformAndSplitSales pipeline.

7. Set up a send port with for example a filter on the name of the Receiveport you've just set up.

8. Drop the example message that's part of the solution!

Final thoughts

This example didn't really take much more than an hour to put together. I do however realize that this is a simplified incoming message and that in the case of a more complex message we'd get a messy XSLT stylesheet to maintain. It's also important to remember that using this method with XSLT transformation means that we will load the entire incoming Xml document into memory, so when we're dealing with bigger Xml documents we'll have manipulate the incoming message using other techniques.

Hope this is useful for someone!

I've blogged about the Visual Studio extension XPathmaniabefore. It's a very simple little tool that lets you write and test XPath inside of Visual Studio 2005. No big deal if you already have tools like XMLSpy or XML Notepad but still. I like not having to start another application, opening the XML document I'm working with and so on. Doing stuff inside of Visual Studio just feels right and saves some time anyway.

One of the last episodes of dnrTV hosted Dom Demsak (Don XML), the creator of XPathmania. The show is 30% about XPathmania and 70% about XPath and XPath syntax in general. It's kind of basic XPath but I think it can be useful for someone who feels they haven't got full control of the language.

During the show they touch on XML namespaces and XML default namespaces. However they don't really explain the difference between them and how it effects the document. Something that's sad as I feel that XML namespaces (and especially default namespaces) is something that most people haven't fully understood.

Anyway, if you got some spare time watch it or forward it someone you think should watch it. ;)

I've often been told that one of the biggest mistakes people make when implementing a service oriented architecture is that they don't re-architect their current architecture to become service oriented. I've never really understood what they meant by that until I read this article. SOA is not about adding a service based call to expose your current procedure calls as a service - it's so much more. It's about enabling ease of change and to create a more agile architecture.

Each of these assumptions exist in a Remote Procedure Call. They are forms of coupling, pure and simple. They fly in the face of SOA. >>

What do you think? Does it make sense in your world?

I've written about the ESB concept before and what I think an ESB architecture is. In the posts comments there were some discussion about if BizTalk is an ESB or not. And if not - why not?

I think this article does a great job in explaining and discussing this subject. Basically it says that the main reason for BizTalk not qualifying as an ESB today (I know about the ESB Guidance - we get there ;)) is because of it's "all-or-nothing" packaging. What that means is that it's different functionality can not be separately deployed a cross a bus structure. For example; the scenario of having the transformation functionality in one place and the routing functionality in another just isn't possible in today's architecture. Today you install the full product in one place.

I think I've personally have learnt to live with this limitation. On the other hand I can see that the possibility of splitting parts up definitely changes things as the possibility of reuse and single point of failure problems etc.

Is BizTalk going the ESB way?

I still haven't had as much time as I'd like to examine the ESB Guidance but I look forward to see how they worked around the problem described above.

Just looking at this architecture image shows that they've split things up in a new way and that each part is accessible trough services - nice! Could this be the future architecture of BizTalk server? What do you think?

I'll try and install the ESB project as soon as I get some more time on my hands. In the mean time I'd love some tips and comments on articles and other reading on the experiences of the ESB Guidance project.

Ordered Delivery is a great feature of BizTalk 2006. The problem is however that as soon as one introduce a orchestration in the process it doesn't work. The Ordered Delivery option on ports in BizTalk 2006 ensures two things today:

1. Messages are published to the MessageBox in the same order as they were consumed by the Receive Port.
2. Messages are consumed from the MessageB0ox and published to the end destination by the send port in the same order as they were persisted in the MessageBox.

What's the problem?

Fine. But if we introduce an orchestration each message being persisted in the the MessageBox will start it's own orchestration instance. These instances could (and in many cases will) finish in different order than they started. This means that we loose the correct order of the messages one they are persisted back to the MessageBox from the orchestration instance.

The above "limitation" is well known and the solution to this has always been a Sequential Orchestration pattern (also called Singleton Orchestration). Basically this correlates messages from for example a specific Receive Port of a particular Message Type. This ensures that only one instance of the orchestration will be started and we can keep the order of the messages. We've experienced a lot of problems using this this kind of solution for Ordered Delivery - all from Zombies to them using huge amounts of memory and so on.

Microsoft's new solution

Now .gif)it finally looks like this problem is about to be solved! Microsoft released a white paper recently (download it here - found it via Richard Seroter’s blog) to solve these problems in a other way. It basically means that a pipeline will stamp the messages with a sequence number as they enter BizTalk (the messages are then in order as we're then using Ordered Delivery on the port). Then one can have as many orchestration as one like processing these messages and publish them back to the MessageBox (now possibly out of order but with a sequence number stamped on them). The last business orchestration will set a "destination" property that will route the message to a resequence orchestration. This orchestration resequences the messages and decides by checking the last sequence number it sent out if the current message should be sent out or put back on queue.

Conclusion

My biggest concern in this solution is that is still based on a singleton. We've had cases where where the send procedure been extremely slow (for example when we used SOAP and had a slow Web Service in the other end), then the orchestration has built up in memory as it queued messages internally. However it looks like this solution is well thought through with a "flush queue" functionality, stop message, some ideas on how to handle errors (remember that if the singleton fails and you don't handle this your processes stop) and so on.

Read it and please let me know what you think!

cXML (commerce eXtensible Markup Language) is a XML based standard for communication of data related to electronic commerce. The problem from a BizTalk perspective is that they don't publish any XML schemas (XSD), only Document Type Definition (DTD).

When trying to generate a schema based on a DTD using the functionality in BizTalk (via Add Generated Items) one ends up with a schema split of three files that really doesn't make any sense (XmlSpy doesn't do a very good job either ...). So after a while I found Nick Heppleston schema repository! After some tweaking I actually had a cXML Order schema in the version I was looking for! Thanks Nick!

The next set of problems was to handle the lack of XML namespace and the DOCTYPE declaration that messages validating against DTD carries on top.

<div><span style="color: #0000FF; "><?</span><span style="color: #FF00FF; ">xml version="1.0" standalone="no"</span><span style="color: #0000FF; ">?></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "><!</span><span style="color: #FF00FF; ">DOCTYPE cXML SYSTEM "http://xml.cxml.org/schemas/cXML/1.2.014/cXML.dtd"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "><</span><span style="color: #800000; ">cXML </span><span style="color: #FF0000; ">xml:lang</span><span style="color: #0000FF; ">="en-US"</span><span style="color: #FF0000; "> payloadID</span><span style="color: #0000FF; ">="2007117.25919@Contempus"</span><span style="color: #FF0000; "> timestamp</span><span style="color: #0000FF; ">="2007-11-07T11:06:16+01:00"</span><span style="color: #0000FF; ">></span></div>

To handel these two issues I set up a receive pipeline that looked like the one below.

Remove the DOCTYPE declaration

First I created a pipeline component to remove the DOCTYPE node. It's simple code using regular expression to find the DOCTYPE node, replace it with an empty string and return the message.

<div><span style="color: #0000FF; ">public</span><span style="color: #000000; "> IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
{
   </span><span style="color: #0000FF; ">string</span><span style="color: #000000; "> messageString </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> StreamReader(inmsg.BodyPart.Data).ReadToEnd();
   Regex doctypePattern </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> Regex(</span><span style="color: #000000; ">"</span><span style="color: #000000; "><!DOCTYPE.+?></span><span style="color: #000000; ">"</span><span style="color: #000000; ">);
   messageString </span><span style="color: #000000; ">=</span><span style="color: #000000; "> doctypePattern.Replace(messageString, </span><span style="color: #0000FF; ">string</span><span style="color: #000000; ">.Empty);
   MemoryStream memStream </span><span style="color: #000000; ">=</span><span style="color: #000000; "> </span><span style="color: #0000FF; ">new</span><span style="color: #000000; "> MemoryStream();
   </span><span style="color: #0000FF; ">byte</span><span style="color: #000000; ">[] data </span><span style="color: #000000; ">=</span><span style="color: #000000; "> Encoding.UTF8.GetBytes(messageString);
   memStream.Write(data, </span><span style="color: #000000; ">0</span><span style="color: #000000; ">, data.Length);
   memStream.Seek(</span><span style="color: #000000; ">0</span><span style="color: #000000; ">, SeekOrigin.Begin);
   inmsg.BodyPart.Data </span><span style="color: #000000; ">=</span><span style="color: #000000; "> memStream;
   </span><span style="color: #0000FF; ">return</span><span style="color: #000000; "> inmsg;
}</span></div>

Set an XML namespace

Secondly I used Richard Seroter's post on how to change the SetNSForMsg component to add a XML namespace. That's the second component showing in the decode stage of the pipeline.

Arrow number 3 shows how the SetMsgNS exposes a property that allows us to set the namespace that we can configure per pipeline. In this case I've set it to http://schemas.modhul.com/cXML/1.2.014/OrderRequest__ which is the namespace of the cXML schema I'm currently working agains.

In the end we'll have a message with the following declaration and root node.

<div><span style="color: #0000FF; "><?</span><span style="color: #FF00FF; ">xml version="1.0" encoding="utf-16" standalone="no"</span><span style="color: #0000FF; ">?></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "><</span><span style="color: #800000; ">cXML </span><span style="color: #FF0000; ">xml:lang</span><span style="color: #0000FF; ">="en-US"</span><span style="color: #FF0000; "> payloadID</span><span style="color: #0000FF; ">="2007117.25919@Contempus"</span><span style="color: #FF0000; "> xmlns</span><span style="color: #0000FF; ">="http://schemas.modhul.com/cXML/1.2.014/OrderRequest"</span><span style="color: #FF0000; "> timestamp</span><span style="color: #0000FF; ">="2007-11-07T11:06:16+01:00"</span><span style="color: #0000FF; ">></span></div>

Now we're ready to start mapping!

Way to often I get a request to deliver a message from BizTalk without any XML namespace. Comments like the one below aren't that rare.

>> Why do you put those "ns0" I front of every tag!? We can't read the XML when it's written like that! All our XPath and XSLT seems to fail. >>>> We don't want _****_ ... We expect the messages from you to be like _****_ ... >>>> Is that some BizTalk specific? >>

I always try to explain what XML namespaces are, and why it's a good idea to use them (especially when it comes to versioning of messages). Sometimes it's just impossible to get people to understand the advantages of using it and to persuade them to change their solutions to handle XML namespaces. It's in these cases it'll be up to the implementation in BizTalk to remove the namespace.

How to remove XML namespace

There are a couple of ways of achieving this, we can use .NET code that we call in a pipeline or in an orchestration. But we can also handle this using XSL - and that's what I'll show in this post. The XSL stylesheet below will remove all XML namespaces while transforming the message. Basically it just copies the nodes, attributes and it's values using the local-name() function to ignore the XML namespaces.

<div><span style="color: #0000ff"><?</span><span style="color: #ff00ff">xml version="1.0" encoding="utf-8"</span><span style="color: #0000ff">?></span><span style="color: #000000">

</span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:stylesheet </span><span style="color: #ff0000">version</span><span style="color: #0000ff">="1.0"</span><span style="color: #ff0000">
    xmlns:xsl</span><span style="color: #0000ff">="http://www.w3.org/1999/XSL/Transform"</span><span style="color: #0000ff">></span><span style="color: #000000">

    </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:template </span><span style="color: #ff0000">match</span><span style="color: #0000ff">="/"</span><span style="color: #0000ff">></span><span style="color: #000000">
        </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:copy</span><span style="color: #0000ff">></span><span style="color: #000000">
            </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:apply-templates </span><span style="color: #0000ff">/></span><span style="color: #000000">
        </span><span style="color: #0000ff"></</span><span style="color: #800000">xsl:copy</span><span style="color: #0000ff">></span><span style="color: #000000">
    </span><span style="color: #0000ff"></</span><span style="color: #800000">xsl:template</span><span style="color: #0000ff">></span><span style="color: #000000">

    </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:template </span><span style="color: #ff0000">match</span><span style="color: #0000ff">="*"</span><span style="color: #0000ff">></span><span style="color: #000000">
        </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:element </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="{local-name()}"</span><span style="color: #0000ff">></span><span style="color: #000000">
            </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:apply-templates </span><span style="color: #ff0000">select</span><span style="color: #0000ff">="@* | node()"</span><span style="color: #ff0000"> </span><span style="color: #0000ff">/></span><span style="color: #000000">
        </span><span style="color: #0000ff"></</span><span style="color: #800000">xsl:element</span><span style="color: #0000ff">></span><span style="color: #000000">
    </span><span style="color: #0000ff"></</span><span style="color: #800000">xsl:template</span><span style="color: #0000ff">></span><span style="color: #000000">

    </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:template </span><span style="color: #ff0000">match</span><span style="color: #0000ff">="@*"</span><span style="color: #0000ff">></span><span style="color: #000000">
        </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:attribute </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="{local-name()}"</span><span style="color: #0000ff">></span><span style="color: #000000">
            </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:value-of </span><span style="color: #ff0000">select</span><span style="color: #0000ff">="."</span><span style="color: #ff0000"></span><span style="color: #0000ff">/></span><span style="color: #000000"></span>
        <span style="color: #0000ff"></</span><span style="color: #800000">xsl:attribute</span><span style="color: #0000ff">></span><span style="color: #000000">
    </span><span style="color: #0000ff"></</span><span style="color: #800000">xsl:template</span><span style="color: #0000ff">></span><span style="color: #000000">

    </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:template </span><span style="color: #ff0000">match</span><span style="color: #0000ff">="text() | processing-instruction() | comment()"</span><span style="color: #0000ff">></span><span style="color: #000000">
        </span><span style="color: #0000ff"><</span><span style="color: #800000">xsl:copy </span><span style="color: #0000ff">/></span><span style="color: #000000">
    </span><span style="color: #0000ff"></</span><span style="color: #800000">xsl:template</span><span style="color: #0000ff">></span><span style="color: #000000">

</span><span style="color: #0000ff"></</span><span style="color: #800000">xsl:stylesheet</span><span style="color: #0000ff">></span></div>

Apply the XSL transformation

We can use this XSL stylesheet in a ordinary BizTalk map on the send port using the Custom XSL Path property in the BizTalk Mapper. The result is that the XSL we usually generate in the mapping tool will be overridden by our own XSL stylesheet. The figure below shows how we use the property windows of the grid in the BizTalk Mapper to set the property and point the Mapper to our XSLT document.

But what if we already have a map on the send port and it's that already transformed message we like to remove the namespace from? One possibility is to use the XSLT Transform pipeline component that comes with the BizTalk 2006 SDK. It's usually located at C:\Program Files\Microsoft BizTalk Server 2006\SDK\Samples\Pipelines\XslTransformComponent\XslTransform on your development machine. I've written about this sample component before here were I used it another scenario.

The figure below shows how we use the property windows of the XSLT Transform Component in a pipeline in the Pipeline designer tool to set the path to our XSLT stylesheet.

Final thoughts

The XSLT Transform component is far from perfect and the obvious problem is of course that the component loads the whole message into memory using the XmlDocument class to read the message. That means that this solution isn't for those scenarios where you'll have huge messages coming in by the thousands. But for those cases where you have normal sized messages and you have a idea of the traffic you receive, it's a quick and easy solution.

Any comments on the pro and cons on this solution and how you usually solve this scenario will be appreciated!

>> **UPDATE #1: **Make sure you don't miss [Johan Hedbergs solution](http://blogical.se/blogs/johan/archive/2008/01/07/removing-xml-namespace-in-a-pipeline-component.aspx) to this problem. Basically he solved it using the [Microsoft.BizTalk.Streaming.dll](http://technet.microsoft.com/en-us/library/microsoft.biztalk.streaming.aspx) which will give you better memory management. >>

>> **UPDATE #2: **Based on the comments to this post I've posted an "updated" post [here](http://www.richardhallgren.com/removing-xml-namespaces-revisit/) solving this using ordinary mapping and defining a schema without a XML namespace - easier and more correct. >>