Using ESB 2.1 with HL7 messages

Using ESB 2.1 with HL7 messages

I assume at this point you have your HL7 messages and its Message header and other schemas deployed in BizTalk. The scenario discussed here is where you receive a HL7 message and you return an ACK back in response to sender of message and send the message to MSMQ.

Using MLLP adapter Or ESB On-Ramp Services

In both cases, we are going to use Request-Response port because that is most typical HL7 message interchange scenario. As part of the sample I created an itinerary which will route the message to MSMQ based on the type of message. Also I have used BRE to resolve itinerary. The following steps I followed to make it work:

Create a new Receive pipeline

This most important step is to create a Receive Pipeline using ESB pipelines and HL7 Disassembler pipeline because none of the off the shelf ESB pipeline uses HL7 Disassembler component. Design the receive pipeline as shown in diagram below:

There are two other components used in addition to HL7 Disassembler component, ESB Itinerary Select, and ESB Dispatcher. These two components are required for ESB to select Itinerary and then execute the itinerary.

               

Create a new Send pipeline

Similar to receive pipeline, create a send pipeline which we will use on Receive port response side. Notice the two ESB pipeline components ESB and ESB Itinerary Cache which are in addition to HL7 Assembler component.

Create an Itinerary Select Rule

Create Itinerary Select Rules which is used by ESB Itinerary Select component to load the itinerary in message context. Please see a sample of Rule. You can use ESB Vocabulary to check for message type and set the required itinerary name and version.

 

Please note the OR condition is required otherwise your condition will not work. The reason for this is HL7 Disassembler component generates two messages if you have configured ACK and this rule will be executed for each of those messages and if you don’t have condition for ACK message then itinerary selection will fail and you will get an error like:

Reason: Procedure or function 'Itinerary_getitinerary' expects parameter '@name', which was not supplied. 

You might now question why do we have Itinerary Select Pipeline component in receive pipeline after HL7 Disassembler component and create our rule based on some other properties rather than message type?

The question is valid and ideally that’s where we should have it unless you want to use message context properties to create rules, however, this approach with HL7 Disassemble component does not work. The problem is when it generates ACK message it does not populate all the original message context properties and as a result your itinerary selection fails.

Create an End Point Set Rule

In this policy, we set the End point of message based on its message type. Since we want to send it to MSMQ we will set the end point accordingly, see the rule below:

Please note, there is no off-the-shelf MSMQ adapter provider, you can follow this article to create this, it’s very simple or you can use some other end point:

http://vikasbhardwaj15.blogspot.com/2013/05/biztalk-esb-toolkit-custom.html

Setup Receive Location and Dynamic Send Port

MLLP Recieve Port

For MLLP adapter, setup a request response MLLP port and location and use your Receive pipeline on Receive pipeline properties as follows:

 

Note you use Itinerary Select Policy here for ESB load itinerary dynamically

There is no specific configuration required for send side.

WCF Generic Receive Port

For WCF Generic On-Ramp, use the ESB.ItineraryServices.Generic.Response.WCF receive location under Microsoft.Practices.ESB application and change it’s receive and send pipeline to your newly created pipelines and configure the receive side as shown above for MLLP.

Dynamic Send Port

Create the Dynamic send port as required by Itinerary offramp, setting filters for servicename, servicestate and servicetype, isrequestresponse.

 

Create Itinerary

Now the last part, Create your itinerary as it’s a simple scenario to route messages, I created a simple itinerary which takes message using On-Ramp one of your receive port (MLLP or OnRamp.Itinerary.Response) resolves the endpoint using our BRE policy using BRE resolver and routes the message accordingly. Now export and deploy your itinerary to Itinerary Db. Make sure you keep the name same as you have it in your Itinerary select rule.

BizTalk ESB Toolkit Custom AdapterProvider and Endpoint Configuration

In my current project assignment, I needed MSMQ AdapterProvider for BizTalk ESB Toolkit and as usual before implementing myself I first searched it on bing and found the link of ESB extension which contains not only MSMQ adapter provider but also others adapter providers. The ESB extension is available on codeplex for ESB 2.0. I started browing the source code of the MSMQ adapter provider and realized that due to some issue MSMQ adapter endpoint configuration are not set if we use BaseAdapterProvider implementation and as a result certain methods were re-implemented in MSMQ provider. In order to find the root cause I started writing a new adapter provider for MSMQ. I thought of documenting this here. Following are the steps you need to follow to create the adapter provider, the amount of code you write is nothing, its all implemented in BaseAdapterProvider. Its all about configuration.

1. Create a new C# project and add a reference to Microsoft.Practices.ESB.Adapter assembly which you can find in Microsoft ESB ToolKit 2.1\bin folder.
2. Created a new class AdapterProvider and extended it from BaseAdapterProvider. BaseAdapterProvider class is abstract implementation of IAdapterProvider interface and provides all basic functionality.
3. Override property AdapterName and return adapter name from it e.g. MSMQ
4. Sign and Build your project and install the assembly in GAC
5. Register your new adapter provider in ESB.Config file as follows:  adapterProvider name="MSMQ" type="ESB.Adapter.MSMQ.AdapterProvider, ESB.Adapter.MSMQ, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8c284297668e995a" moniker="MSMQ"
6. At first you think that's it and now you can see your adapter provider in your resolver under the TransportName.
7. If you just need the default basic properties on your adapter provider e.g. Transport Name, Transport Location then you do not need anything else. However if you click now on Endpoint Configuration option then you get an error as shown below:
   

WCF ServiceModelReg Error on Windows 8

If you try to activate WCF model on Windows 8 environment by running ServiceModelReg -r -y command on C:\Windows\Microsoft.Net\Framework64\v4.0.30319, then you will following error:


[Error]This tool is not supported on this version of Windows. Administrators should instead install/uninstall Windows Communication Foundation features using the 'Turn Windows Features On/Off' dialog, or the dism command line tool.

Once you go to Program and Features and Turn Windows Features On/Off, then Expand .NET Framework 4.5 Advanced Services ->WCF Services and select Http Activation as shown below:

This will register WCF service model with .Net Framework 4.5
 

Understanding BizTalk ESB ToolKit

Understanding BizTalk ESB ToolKit

I was going through various documents to understand BizTalk ESB Toolkit 2.1 architecture to know how it works and how can I put this in simple words for people to start with. Lets try to describe it. Please note before reading this article you should have some basic understanding of ESB

What is BizTalk ESB Toolkit

BizTalk ESB toolkit is an add-on installation to BizTalk Server and provides following components to work with:

- Itinerary

- Receive and Send Pipeline 

- Pipeline Components

- Adapter and Resolver Framework

- Exception Management

- On Ramp Web Services

- Core Web Services - Resolver, Operations, Exception, UDDI and Transformation Services

Itinerary - An itinerary is a set of steps, where each step performs an action e.g. message transformation, routing, orchestration execution. 

The next three components i.e. Pipelines, Pipeline Components, and Adapter & Resolver Framework together provide dynamic resolution of End Points, Transformation maps and Itinerary based on message context.

Exception Management - provides a framework which can be used not only in BizTalk applications but also in other non BizTalk application to log exceptions at one place.

On Ramp Web Services  - There are number of services exposed once you configure ESB toolkit and primary purpose of these services is allow consumers to submit message to ESB over http protocol. The receive location corresponding to these web services in BizTalk are referred to as On-Ramp. However your On-Ramp Receive locations can also receive messages from other protocol e.g. using File Adapter.

Core Web Services - Each of these services expose specific functionality as follows:

 - Resolver and Transformation - These services extends the dynamic resolution and dynamic transformation functionality to consumer over web service. Behind the scene these work based on Adapter & Resolver Framework only.

- Operations - Operation web service expose BizTalk operation service and has handful of operations to browse various BizTalk artifacts. This is subset of BizTalk OperationOM dll

- Exception - Exception Service allow consumers to save exception to exception store

- UDDI - UDDI service expose UDDI3 feature as web service which can used to search for registered services etc.

How all these components fit together

See the diagram below: It shows various components of ESB and how they interact with each other. Each Marked shape in the diagram is described below:

1. Consumer sends a message to ESB using one of the On-Ramp service. There are 6 On-Ramp services, 3 for one way and 3 for two-way operations. And each operation is supported over ASMX, WCF and WCF Generic. The difference between three different service types is explained in point 3

2. On-Ramp receive location receives the message and executes the pipeline.
3. Depending on which On-Ramp service has been used for submitting the message, There are three ways an itinerary is identified for an incoming message:
a - ASMX Service - Whole itinerary is part of message sent as part of message soap header. In this case consumer sends itinerary xml within message soap header while calling the On-Ramp service. This is does using ESB.ItineraryServices/ProcessItinerary.asmx On-Ramp service for one way operation and ESB.ItineraryServices.Response/ProcessItinerary.asmx for two way operation . In this case while developing itinerary in visual studio, you chose the option of exporting itinerary to xml rather than itineray store
b -  WCF Service - Itinerary is stored in Itinerary store database in ESB environment, and message sender sends the itinerary name, and version along with message. The consumer in this case use ESB.ItineraryServices.WCF/ProcessItinerary.svc On-Ramp service for one way operation and ESB.ItineraryServices.Response.WCF/ProcessItinerary.svc for two way operation.
c - WCF Generic - No itinerary detail is sent along with message in this case, consumer only passes message with not details on how the message will be processed. On ESB side, itinerary selector resolves the itinerary using resolver framework based on message context and load the itinerary from itinerary store for execution. This is rececommanded way, the other two ways are available for backward compatibility with ESB toolkit 1.0. The consumer uses ESB.ItineraryServices.Generic.WCF/ProcessItinerary.svc for one way operation and ESB.ItineraryServices.Generic.Response.WCF/ProcessItinerary.svc for two way operation.

4. Once Itinerary is identified and loaded from message or itinerary store, itinerary selector component  attaches it to the message context. From here onwards, itinerary is part of message context.

5. Based on the steps defined in itinerary, the ESB Dispatcher component of pipeline executes either the Messaging Itinerary service and performs routing/transformation within pipeline component or
6, 7. It publishes the message to Message Box to be picked up by either Orchestration Service or send port (Off-Ramp).

Messaging Itinerary and Orchestration Itinerary Service are two ways you can route/transform messages in ESB dynamically. Messaging Itinerary gets executed within pipeline component scope while orchestration itinerary service is nothing but an execution of an orchestration. So use orchestration service in case you want to execute some business process in addition to routing and transformation. In most other cases Messaging Itinerary Service should suffice.

8. Send Ports (Off-Ramp) also use many ESB pipelines while sending the messages. These sends ports are dynamic send ports, either one-way, two-way, forwarder etc. The three key points to remember while configuring Send Ports and using them in itinerary are:
 - Create them Dynamic
 - Use Filter having minimum following properties
       - Microsoft.Practices.ESB.Schema.ServiceName = Any Name you want to give here, can be any string
       - Microsoft.Practices.ESB.Schema.ServiceType = Messaging, set it to "Messaging" indicating that this port is being used by Messaging Itinerary Service
       - Microsoft.Practices.ESB.Schema.ServiceState = Pending, set it "Pending" always
       - Microsoft.Practices.ESB.Schema.IsRequestResponse = True/False based on whether port is one-way or two-way.
  - Use the appropriate pipeline based on requirement. There are number of pipelines to choose from.

9. Once you configure ESB, Microsoft.Practices.ESB application in BizTalk Admin console contains a send port ALL.Exceptions which consumes all exceptions message from Message Box and writes them to EsbExceptionDb which is used by ESB.Portal application to show faults or generate alerts. The exception can also be written by calling ExceptionService. In your design, ESB Exception Management can become central repository for all exceptions including those which are outside BizTalk environment applications.

10. Adpater Provider and Resolver Framework - In my view, this is the backbone of ESB architecture which is used every where, the best part of it is that it's very extensible.

In simple terms resolver framework helps you resolve the maps name, send port destination details (end points), itinerary name. There are different type of resolver, BRE, STATIC, UDDI etc. The purpose of each resolver is to resolve one of the above three things based on your scenario. e.g. you can store all your endpoint configuration in UDDI and use UDDI resolver to retrieve all that information at run time based on the message context. Consider each different types of resolver as your different configuration store. So each Resolver gets the details based on selected resolver and returns the data in form of a dictionary. That's all it does. 

A resolver is called by either pipeline component like Itinerary Selector, ESB Dispatcher or Messaging/Orchestration Services. After the resolution is done, depending on what is being resolved one of following happens:
a - If its an end point resolution (usually you will do in off Ramp, in routing service), Adapter provider sets adapter properties for dynamic routing. There are different adapter provider for specific adapter, and each of them set required properties of the adapter for dynamic routing.
b - If its for transformation resolution, then transformation service will execute the resolved map as required.
c - If its for itinerary resolution, then itinerary selector component will load the itinerary from itinerary store.

Both Adapter Provider and Resolver can be extended by creating customer adapter provider and resolver by implementing IAdapterProvider and IResolver interface and registering them Esb.Config respectively.

11. ESB Portal is part of ESB Samples with source code. This can be used as a start for managing exceptions within ESB. This allows to generate alert, message correction or message re-submission as well. Also you are free to extend its capability to your need.

12. The core web services as mentioned above can be used by consumers to use ESB feature of dynamic resolution, transformation even in non BizTalk environment. Some of other services like Operations, UDDI gives administrative capability for BizTalk and UDDI as well. The exception service as mentioned above can be used to submit exception to EsbExceptionDb.

I hope it makes it easy to understand ESB Toolkit and its various components.

BizTalk Server ESB Toolkit Service ESB.ItineraryServices.Generic.Response.WCF Fails to browse

If you browse this service after installing and configuring ESB toolkit then you receive one of the error depending on if you are using BizTalk 2010 or BizTalk 2013:

BizTalk 2010

Error 855000: Could not find schema file for SOAP header 'ItineraryDescription' at path C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Web\ESB.ItineraryServices.Generic.Response.WCF\App_Data\ItineraryDescription.xsd

BizTalk 2013

An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is:
System.ArgumentException: The value could not be added to the collection, as the collection already contains an item of the same type: 'Microsoft.Practices.ESB.ServiceModel.Helpers.SoapHeaderMetadataBehavior'. This collection only supports one instance of each type.

Cause

By definition, Generic On-Ramp services are meant for not passing the itinerary details with the message and so they don't need to have ItineraryDescription. This is how One Way On-Ramp service also works. However this ESB.ItineraryServices.Generic.Response.WCF On-Ramp service expects to have soap header with Itinerary description as per its configuration but from a deployment perspective there is no ItineraryDescription.xsd in the folder and as a result this error is returned.

Solution

The solution to this problem is to disable soapHeaderMetadata in Web.Config file so that WCF does not try to load the SoapHeader information. So within the serviceBehaviors node change soapHeaderMetadata node attribute enabled=false as follows and recycle the IIS application pool:

soapHeaderMetadata enabled="false"

BizTalk ESB Toolkit Sample Service ESB NorthAmericanServices Fails to execute

If you try to call ESB.NorthAmericanServices/CustomerOrder.asmx web service which is part of Dynamic Resolution Sample provided as part of ESB ToolKit 2.1, then most likely you will get the following error:

There was a failure executing the receive pipeline: "GlobalBank.ESB.DynamicResolution.Pipelines.ESBReceiveXML, GlobalBank.ESB.DynamicResolution.Pipelines, Version=2.1.0.0, Culture=neutral, PublicKeyToken=e457dd48ca33cd6f" Source: "ESB Dispatcher Disassemble" Receive Port: "DynamicResolutionReqResp" URI: "/ESB.NorthAmericanServices/CustomerOrder.asmx" Reason: The document specification from assembly failed to load. Verify the schema for this document specification is deployed and is in the Global Assembly Cache.

Cause

The cause of this issue is the code of ESB.NorthAmericanServices/CustomerOrder.asmx refers to the old dll GlobalBank.ESB.DynamicResolution.Schemas V2.0.0.0.

Most likely Microsoft forgot to regenerate the service code after updating their assemblies to 2.1.0.0 version

Solution

The fix for this solution is to go to ESB.NorthAmericanServces\App_Code\CustomerOrder.asmx.cs file and go to submitOrder method code and change the following line of code:

string bodyTypeAssemblyQualifiedName = "GlobalBank.ESB.DynamicResolution.Schemas.NAOrderDoc, GlobalBank.ESB.DynamicResolu" +

"tion.Schemas, Version=2.0.0.0, Culture=neutral, PublicKeyToken=e457dd48ca33cd6f";

                To

string bodyTypeAssemblyQualifiedName = "GlobalBank.ESB.DynamicResolution.Schemas.NAOrderDoc, GlobalBank.ESB.DynamicResolu" +

"tion.Schemas, Version=2.1.0.0, Culture=neutral, PublicKeyToken=e457dd48ca33cd6f";

Error - The itinerary header could not be found Or There is an error in XML document (1, 2).

Once you resolve the above error then you may get this error depending on how are you calling this service:

NorthAmericanServices is nothing but a re-routing service to CanadianServices and that's why there is no corresponding orchestration.

If you look at closely at DynamicResolutionReqResp_SOAP receive location, it uses a ESBReceiveXML pipeline which uses ESB Dispatcher component to resolve the end points of canadian web service from BRE resolver and then it routes the service call to Canadian service. 

From here the call goes to DynamicResolutionSolicitResp send port, due to its filter on DynamicResolutionReqResp receive port. 

You may receive error "There is an error in XML document (1, 2)." This error occurs on DynamicResolutionReqResp receive port because it tries to route the request to Canadian web service while it receive request for NA web service, the schema for both are different. In order to resolve this issue, You need to make sure you have Inbound and Outbound Map on receive port to convert NA request to CN request on Inbound side and to convert CN response to NA response on outbound side. These maps are already present in Dynamic Resolution sample of ESB

Or
You may also receive "the itinerary header could not be found" on DynamicResolutionSolicitResp send port. Because this port uses ItinerarySendPassthrough pipeline it expects to have itineraryheader to be present in message context and if it does not find it throws this error.

If you call this service from an itinerary it will work on this send port. However in order to call the NorthAmericanServices directly from a .net app, you need to create another two-way send port which use XmlTransmit and XmlReceive pipelines.

BizTalk ESB Portal Initial Setup Issue

While configuring ESB Portal on my machine I encountered few issues, few of those issues fixes are listed here:

1. The remote server returned an error: (404) not Found

This error details are logged in Eventview, the snippet of the error is shown below:

Application information: Application domain: /LM/W3SVC/1/ROOT/ESB.Portal-1-130089573102692686 Trust level: Full Application Virtual Path: /ESB.Portal Application Path: C:\Projects\Microsoft.Practices.ESB\ESBSource\Source\Samples\Management Portal\ESB.Portal\ Machine name: Process information: Process ID: 10524 Process name: w3wp.exe Account name: NT AUTHORITY\NETWORK SERVICE Exception information: Exception type: WebException Exception message: The remote server returned an error: (404) Not Found. Request information: Request URL: http://localhost/ESB.Portal/default.aspx Request path: /ESB.Portal/default.aspx User host address: 127.0.0.1 User: Is authenticated: True Authentication Type: Negotiate Thread account name: NT AUTHORITY\NETWORK SERVICE Thread information: Thread ID: 7 Thread account name: NT AUTHORITY\NETWORK SERVICE Is impersonating: False Stack trace: at System.Net.HttpWebRequest.GetResponse()

at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)

Possible Causes

- Database connectivity - Esb Portal Web.Config does not have the correct database settings.
- Underlying web service connectivity issues - Esb Portal uses few web services deployed on IIS, e.g. Esb.BAM.Service, Esb.Exceptions.Service, Esb.BizTalkOperationService etc and if there is any issue with connectivity to these services, then it results in above errors.

Solution

• Check the web.config file for correct database connection strings, if you are testing locally you can replace them from following:

Check now if Esb portal works fine. If not then try and check each of above (under cause) mentioned web service if they are running fine or not, if not and you are facing an error like:

“The requested content appears to be script and will not be served by the static file handler.”

Try following steps:

• In a command prompt, change to the C:\Windows\Microsoft .NET\Framework\v3.0\Windows Communication Foundation\ directory, and then run ServiceModelReg.exe /x /i.

Note: This will ensure that WCF support is fully installed and available.

• Go to C:\Windows\Microsoft .NET\Framework\v4.0.30319\ and run aspnet_regiis –i

• iisreset

2. Index (zero based) must be greater than or equal to zero and less than the size of the argument list.

This error usually comes from BizTalk operation Service which is accessed by Esb Portal. The following error details is from EventViewer.

Application information: Application domain: /LM/W3SVC/1/ROOT/ESB.Portal-3-130089589367504127 Trust level: Full Application Virtual Path: /ESB.Portal Application Path: C:\Projects\Microsoft.Practices.ESB\ESBSource\Source\Samples\Management Portal\ESB.Portal\ Machine name: Process information: Process ID: 8828 Process name: w3wp.exe Account name: NT AUTHORITY\NETWORK SERVICE Exception information: Exception type: SoapException Exception message: An unexpected error occurred during initialization of the BizTalk Operations Service. ---> Index (zero based) must be greater than or equal to zero and less than the size of the argument list.

Cause

It is an access right issue to retrieve information from BizTalk management database.

Solution

• Check BizTalk operation service application pool user account. This user account should be part of BTS_ADM_USERS Database role for BizTalk Management Database.