MSDTC Log Full Error Resolution on BizTalk Server

 As we know BizTalk is heavily dependent on MSDTC for controlling the transactions e.g. adapters like WCF-SQL, MSMQ adapters span MSDTC transaction for maintaining the transaction between BizTalk MsgBox and other system. The MSDTC service normally works fine without any issues however in case of high concurrent volume, MSTDC transaction log can get full because its default size is 4MB only. In such cases BizTalk adapters starts returning following errors
System.Transactions.TransactionException: The Transaction Manager returned a log full error. (Exception from HRESULT: 0x8004D01A) ---> System.Runtime.InteropServices.COMException: The Transaction Manager returned a log full error. (Exception from HRESULT: 0x8004D01A)

Resolution

In order to fix the issue, perform following steps:

• Stop all BizTalk hosts trying to initiate a transaction with MSDTC on particular server having issues.
• Logon to the server with Admin privilege account and Run dcomcnfg to launch the Component Services Console. In Component Services Go to Local DTC as shown below:
    

• Right Click on Local DTC and go to Properties and Select Logging Tab as shown below:
    

• Increase the capacity from 4MB to the size based on your volume e.g. 50 or 100MB and hit OK. On Pressing OK, It will restart the MSDTC service on the server.

Notes  

• If you are not using Admin account on server and try to follow above steps, MSDTC service will not restart properly and will give error.
  
• After restart of MSDTC service successfully, if you are using MSMQ adapter then make sure you also restart MSMQ service on the server, otherwise even after increasing the log size, and restarting all hosts, you will get an error "Transaction can't be enlisted".
  

 

 

 

 

Publish REST Service from BizTalk Server

Publish REST Service from BizTalk Server 2013 WCF-WebHttp Adapter

BizTalk Server 2013 provides REST Services support and it makes it easy to consume and publish REST services. I saw many example of consuming REST Services in BizTalk, however could not find anything for publishing REST Services on BizTalk. In this article I am going to detail on steps of publishing REST services.

 

For this article I referred to Steve-Jan Wiggers sample which can be found at following location http://code.msdn.microsoft.com/BizTalk-Server-2013-WCF-e3e4a4f9 

This sample talks about consuming a REST service to get airport status. In order to test the sample Steve has a Windows Form based application which invokes a WCF service exposed by BizTalk.

 

I further extended the sample to expose REST service using WCF-WebHttp Adapter and tested the whole scenario with Fiddler and IE. The details steps are as follows:

 

By this time, I assume you have run the above mentioned code sample

 

• Run WCF Publishing Wizard from Visual Studio Tools Menu:

 

 

• From the WCF Publishing Wizard, Select WCF-WebHttp Adapter on Service endpoint option and click Next

 

• Choose default option on AppFabric Connect, as we are not concentrating on ServiceBus right now and click Next
• Choose Create a Request-Response Receive Port and click Next

• Choose IIS publish location and select allow Anonymous access and click next
• The wizard will now display the summary page of the selection
• Click Create to publish the REST Service:

Few points to note after service is published:

• The WCF service wizard does not ask for any BizTalk Orchestration and Schema details Or Service Operation details as it does in other WCF service publishing. At first I thought there is something wrong, however then I realized that for REST services you do not need metadata information so there is no need to define the service operation.
• Service is published in IIS using default app pool of the IIS web site selected on the Wizard, if the app pool of the site does not have required BizTalk access, please change app pool.
• The name of the service is created as Service1.svc, for this sample to work you do not need to change the name.
• Make sure your service app pool does not have any other service running under it, e.g. if in the original sample, you published as WCF service and that service is running under the same account as REST service is running, you will start getting the following error:

"Registering multiple adapter types within the same process is not a supported configuration. For e.g. HTTP and SOAP receive adapters cannot co-exist in the same process"

You can now check the service by running the Service1.svc URL in IE and see if you are getting the service page successfully.

Now check out the BizTalk Receive Location which was created by WCF Service Publishing Wizard. You now need to configure the BizTalk operation you want your REST Service to support. The default operation configuration is not correct and should be changed as shown in below steps:

• Open the Receive Location Transport Type (Adapter) Configuration:

 

• Check out the HTTP Method and URL mapping box, its not what we want here, so we changed it to a GET operation with a URL /status/{airportcode}, where airportcode is a variable which will be mapped with a message context as shown below:

 
• The mapping of variable {airportcode} with message context properties allows BizTalk adapter to write the context property to incoming request and also allows mapped message context property to be used as a replacement while sending the request to a REST service.
• This configuration now allows One REST operation 'GET' with URL http://localhost/.../Service1.svc/status/BNA where BNA is the airportcode

 

If you now browse this URL in IE, then you should see a response similar to the response of http://services.faa.gov/airport/status/SEA?format=xml. Because in BizTalk we are just forwarding the request to faa.gov REST service.

 

You can also use Fiddler to create an REST request as shown below

 

Fiddler Request

 

 

 

In case of GET operation, there is no message body required, which can be ignored completely. However in some operation like POST, you need the message body. In such cases body can be supplied with appropriate content-type in the HTTP header.

 

I hope it helps you.

 

 

Using ESB 2.1 with HL7 messages

Using ESB 2.1 with HL7 messages

I assume at this point you have your HL7 messages and its Message header and other schemas deployed in BizTalk. The scenario discussed here is where you receive a HL7 message and you return an ACK back in response to sender of message and send the message to MSMQ.

Using MLLP adapter Or ESB On-Ramp Services

In both cases, we are going to use Request-Response port because that is most typical HL7 message interchange scenario. As part of the sample I created an itinerary which will route the message to MSMQ based on the type of message. Also I have used BRE to resolve itinerary. The following steps I followed to make it work:

Create a new Receive pipeline

This most important step is to create a Receive Pipeline using ESB pipelines and HL7 Disassembler pipeline because none of the off the shelf ESB pipeline uses HL7 Disassembler component. Design the receive pipeline as shown in diagram below:

There are two other components used in addition to HL7 Disassembler component, ESB Itinerary Select, and ESB Dispatcher. These two components are required for ESB to select Itinerary and then execute the itinerary.

               

Create a new Send pipeline

Similar to receive pipeline, create a send pipeline which we will use on Receive port response side. Notice the two ESB pipeline components ESB and ESB Itinerary Cache which are in addition to HL7 Assembler component.

Create an Itinerary Select Rule

Create Itinerary Select Rules which is used by ESB Itinerary Select component to load the itinerary in message context. Please see a sample of Rule. You can use ESB Vocabulary to check for message type and set the required itinerary name and version.

 

Please note the OR condition is required otherwise your condition will not work. The reason for this is HL7 Disassembler component generates two messages if you have configured ACK and this rule will be executed for each of those messages and if you don’t have condition for ACK message then itinerary selection will fail and you will get an error like:

Reason: Procedure or function 'Itinerary_getitinerary' expects parameter '@name', which was not supplied. 

You might now question why do we have Itinerary Select Pipeline component in receive pipeline after HL7 Disassembler component and create our rule based on some other properties rather than message type?

The question is valid and ideally that’s where we should have it unless you want to use message context properties to create rules, however, this approach with HL7 Disassemble component does not work. The problem is when it generates ACK message it does not populate all the original message context properties and as a result your itinerary selection fails.

Create an End Point Set Rule

In this policy, we set the End point of message based on its message type. Since we want to send it to MSMQ we will set the end point accordingly, see the rule below:

Please note, there is no off-the-shelf MSMQ adapter provider, you can follow this article to create this, it’s very simple or you can use some other end point:

http://vikasbhardwaj15.blogspot.com/2013/05/biztalk-esb-toolkit-custom.html

Setup Receive Location and Dynamic Send Port

MLLP Recieve Port

For MLLP adapter, setup a request response MLLP port and location and use your Receive pipeline on Receive pipeline properties as follows:

 

Note you use Itinerary Select Policy here for ESB load itinerary dynamically

There is no specific configuration required for send side.

WCF Generic Receive Port

For WCF Generic On-Ramp, use the ESB.ItineraryServices.Generic.Response.WCF receive location under Microsoft.Practices.ESB application and change it’s receive and send pipeline to your newly created pipelines and configure the receive side as shown above for MLLP.

Dynamic Send Port

Create the Dynamic send port as required by Itinerary offramp, setting filters for servicename, servicestate and servicetype, isrequestresponse.

 

Create Itinerary

Now the last part, Create your itinerary as it’s a simple scenario to route messages, I created a simple itinerary which takes message using On-Ramp one of your receive port (MLLP or OnRamp.Itinerary.Response) resolves the endpoint using our BRE policy using BRE resolver and routes the message accordingly. Now export and deploy your itinerary to Itinerary Db. Make sure you keep the name same as you have it in your Itinerary select rule.

BizTalk ESB Toolkit Custom AdapterProvider and Endpoint Configuration

In my current project assignment, I needed MSMQ AdapterProvider for BizTalk ESB Toolkit and as usual before implementing myself I first searched it on bing and found the link of ESB extension which contains not only MSMQ adapter provider but also others adapter providers. The ESB extension is available on codeplex for ESB 2.0. I started browing the source code of the MSMQ adapter provider and realized that due to some issue MSMQ adapter endpoint configuration are not set if we use BaseAdapterProvider implementation and as a result certain methods were re-implemented in MSMQ provider. In order to find the root cause I started writing a new adapter provider for MSMQ. I thought of documenting this here. Following are the steps you need to follow to create the adapter provider, the amount of code you write is nothing, its all implemented in BaseAdapterProvider. Its all about configuration.

1. Create a new C# project and add a reference to Microsoft.Practices.ESB.Adapter assembly which you can find in Microsoft ESB ToolKit 2.1\bin folder.
2. Created a new class AdapterProvider and extended it from BaseAdapterProvider. BaseAdapterProvider class is abstract implementation of IAdapterProvider interface and provides all basic functionality.
3. Override property AdapterName and return adapter name from it e.g. MSMQ
4. Sign and Build your project and install the assembly in GAC
5. Register your new adapter provider in ESB.Config file as follows:  adapterProvider name="MSMQ" type="ESB.Adapter.MSMQ.AdapterProvider, ESB.Adapter.MSMQ, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8c284297668e995a" moniker="MSMQ"
6. At first you think that's it and now you can see your adapter provider in your resolver under the TransportName.
7. If you just need the default basic properties on your adapter provider e.g. Transport Name, Transport Location then you do not need anything else. However if you click now on Endpoint Configuration option then you get an error as shown below:
   

WCF ServiceModelReg Error on Windows 8

If you try to activate WCF model on Windows 8 environment by running ServiceModelReg -r -y command on C:\Windows\Microsoft.Net\Framework64\v4.0.30319, then you will following error:


[Error]This tool is not supported on this version of Windows. Administrators should instead install/uninstall Windows Communication Foundation features using the 'Turn Windows Features On/Off' dialog, or the dism command line tool.

Once you go to Program and Features and Turn Windows Features On/Off, then Expand .NET Framework 4.5 Advanced Services ->WCF Services and select Http Activation as shown below:

This will register WCF service model with .Net Framework 4.5
 

Understanding BizTalk ESB ToolKit

Understanding BizTalk ESB ToolKit

I was going through various documents to understand BizTalk ESB Toolkit 2.1 architecture to know how it works and how can I put this in simple words for people to start with. Lets try to describe it. Please note before reading this article you should have some basic understanding of ESB

What is BizTalk ESB Toolkit

BizTalk ESB toolkit is an add-on installation to BizTalk Server and provides following components to work with:

- Itinerary

- Receive and Send Pipeline 

- Pipeline Components

- Adapter and Resolver Framework

- Exception Management

- On Ramp Web Services

- Core Web Services - Resolver, Operations, Exception, UDDI and Transformation Services

Itinerary - An itinerary is a set of steps, where each step performs an action e.g. message transformation, routing, orchestration execution. 

The next three components i.e. Pipelines, Pipeline Components, and Adapter & Resolver Framework together provide dynamic resolution of End Points, Transformation maps and Itinerary based on message context.

Exception Management - provides a framework which can be used not only in BizTalk applications but also in other non BizTalk application to log exceptions at one place.

On Ramp Web Services  - There are number of services exposed once you configure ESB toolkit and primary purpose of these services is allow consumers to submit message to ESB over http protocol. The receive location corresponding to these web services in BizTalk are referred to as On-Ramp. However your On-Ramp Receive locations can also receive messages from other protocol e.g. using File Adapter.

Core Web Services - Each of these services expose specific functionality as follows:

 - Resolver and Transformation - These services extends the dynamic resolution and dynamic transformation functionality to consumer over web service. Behind the scene these work based on Adapter & Resolver Framework only.

- Operations - Operation web service expose BizTalk operation service and has handful of operations to browse various BizTalk artifacts. This is subset of BizTalk OperationOM dll

- Exception - Exception Service allow consumers to save exception to exception store

- UDDI - UDDI service expose UDDI3 feature as web service which can used to search for registered services etc.

How all these components fit together

See the diagram below: It shows various components of ESB and how they interact with each other. Each Marked shape in the diagram is described below:

1. Consumer sends a message to ESB using one of the On-Ramp service. There are 6 On-Ramp services, 3 for one way and 3 for two-way operations. And each operation is supported over ASMX, WCF and WCF Generic. The difference between three different service types is explained in point 3

2. On-Ramp receive location receives the message and executes the pipeline.
3. Depending on which On-Ramp service has been used for submitting the message, There are three ways an itinerary is identified for an incoming message:
a - ASMX Service - Whole itinerary is part of message sent as part of message soap header. In this case consumer sends itinerary xml within message soap header while calling the On-Ramp service. This is does using ESB.ItineraryServices/ProcessItinerary.asmx On-Ramp service for one way operation and ESB.ItineraryServices.Response/ProcessItinerary.asmx for two way operation . In this case while developing itinerary in visual studio, you chose the option of exporting itinerary to xml rather than itineray store
b -  WCF Service - Itinerary is stored in Itinerary store database in ESB environment, and message sender sends the itinerary name, and version along with message. The consumer in this case use ESB.ItineraryServices.WCF/ProcessItinerary.svc On-Ramp service for one way operation and ESB.ItineraryServices.Response.WCF/ProcessItinerary.svc for two way operation.
c - WCF Generic - No itinerary detail is sent along with message in this case, consumer only passes message with not details on how the message will be processed. On ESB side, itinerary selector resolves the itinerary using resolver framework based on message context and load the itinerary from itinerary store for execution. This is rececommanded way, the other two ways are available for backward compatibility with ESB toolkit 1.0. The consumer uses ESB.ItineraryServices.Generic.WCF/ProcessItinerary.svc for one way operation and ESB.ItineraryServices.Generic.Response.WCF/ProcessItinerary.svc for two way operation.

4. Once Itinerary is identified and loaded from message or itinerary store, itinerary selector component  attaches it to the message context. From here onwards, itinerary is part of message context.

5. Based on the steps defined in itinerary, the ESB Dispatcher component of pipeline executes either the Messaging Itinerary service and performs routing/transformation within pipeline component or
6, 7. It publishes the message to Message Box to be picked up by either Orchestration Service or send port (Off-Ramp).

Messaging Itinerary and Orchestration Itinerary Service are two ways you can route/transform messages in ESB dynamically. Messaging Itinerary gets executed within pipeline component scope while orchestration itinerary service is nothing but an execution of an orchestration. So use orchestration service in case you want to execute some business process in addition to routing and transformation. In most other cases Messaging Itinerary Service should suffice.

8. Send Ports (Off-Ramp) also use many ESB pipelines while sending the messages. These sends ports are dynamic send ports, either one-way, two-way, forwarder etc. The three key points to remember while configuring Send Ports and using them in itinerary are:
 - Create them Dynamic
 - Use Filter having minimum following properties
       - Microsoft.Practices.ESB.Schema.ServiceName = Any Name you want to give here, can be any string
       - Microsoft.Practices.ESB.Schema.ServiceType = Messaging, set it to "Messaging" indicating that this port is being used by Messaging Itinerary Service
       - Microsoft.Practices.ESB.Schema.ServiceState = Pending, set it "Pending" always
       - Microsoft.Practices.ESB.Schema.IsRequestResponse = True/False based on whether port is one-way or two-way.
  - Use the appropriate pipeline based on requirement. There are number of pipelines to choose from.

9. Once you configure ESB, Microsoft.Practices.ESB application in BizTalk Admin console contains a send port ALL.Exceptions which consumes all exceptions message from Message Box and writes them to EsbExceptionDb which is used by ESB.Portal application to show faults or generate alerts. The exception can also be written by calling ExceptionService. In your design, ESB Exception Management can become central repository for all exceptions including those which are outside BizTalk environment applications.

10. Adpater Provider and Resolver Framework - In my view, this is the backbone of ESB architecture which is used every where, the best part of it is that it's very extensible.

In simple terms resolver framework helps you resolve the maps name, send port destination details (end points), itinerary name. There are different type of resolver, BRE, STATIC, UDDI etc. The purpose of each resolver is to resolve one of the above three things based on your scenario. e.g. you can store all your endpoint configuration in UDDI and use UDDI resolver to retrieve all that information at run time based on the message context. Consider each different types of resolver as your different configuration store. So each Resolver gets the details based on selected resolver and returns the data in form of a dictionary. That's all it does. 

A resolver is called by either pipeline component like Itinerary Selector, ESB Dispatcher or Messaging/Orchestration Services. After the resolution is done, depending on what is being resolved one of following happens:
a - If its an end point resolution (usually you will do in off Ramp, in routing service), Adapter provider sets adapter properties for dynamic routing. There are different adapter provider for specific adapter, and each of them set required properties of the adapter for dynamic routing.
b - If its for transformation resolution, then transformation service will execute the resolved map as required.
c - If its for itinerary resolution, then itinerary selector component will load the itinerary from itinerary store.

Both Adapter Provider and Resolver can be extended by creating customer adapter provider and resolver by implementing IAdapterProvider and IResolver interface and registering them Esb.Config respectively.

11. ESB Portal is part of ESB Samples with source code. This can be used as a start for managing exceptions within ESB. This allows to generate alert, message correction or message re-submission as well. Also you are free to extend its capability to your need.

12. The core web services as mentioned above can be used by consumers to use ESB feature of dynamic resolution, transformation even in non BizTalk environment. Some of other services like Operations, UDDI gives administrative capability for BizTalk and UDDI as well. The exception service as mentioned above can be used to submit exception to EsbExceptionDb.

I hope it makes it easy to understand ESB Toolkit and its various components.

BizTalk Server ESB Toolkit Service ESB.ItineraryServices.Generic.Response.WCF Fails to browse

If you browse this service after installing and configuring ESB toolkit then you receive one of the error depending on if you are using BizTalk 2010 or BizTalk 2013:

BizTalk 2010

Error 855000: Could not find schema file for SOAP header 'ItineraryDescription' at path C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Web\ESB.ItineraryServices.Generic.Response.WCF\App_Data\ItineraryDescription.xsd

BizTalk 2013

An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is:
System.ArgumentException: The value could not be added to the collection, as the collection already contains an item of the same type: 'Microsoft.Practices.ESB.ServiceModel.Helpers.SoapHeaderMetadataBehavior'. This collection only supports one instance of each type.

Cause

By definition, Generic On-Ramp services are meant for not passing the itinerary details with the message and so they don't need to have ItineraryDescription. This is how One Way On-Ramp service also works. However this ESB.ItineraryServices.Generic.Response.WCF On-Ramp service expects to have soap header with Itinerary description as per its configuration but from a deployment perspective there is no ItineraryDescription.xsd in the folder and as a result this error is returned.

Solution

The solution to this problem is to disable soapHeaderMetadata in Web.Config file so that WCF does not try to load the SoapHeader information. So within the serviceBehaviors node change soapHeaderMetadata node attribute enabled=false as follows and recycle the IIS application pool:

soapHeaderMetadata enabled="false"