Understanding BizTalk ESB ToolKit

Understanding BizTalk ESB ToolKit

I was going through various documents to understand BizTalk ESB Toolkit 2.1 architecture to know how it works and how can I put this in simple words for people to start with. Lets try to describe it. Please note before reading this article you should have some basic understanding of ESB

What is BizTalk ESB Toolkit

BizTalk ESB toolkit is an add-on installation to BizTalk Server and provides following components to work with:

- Itinerary

- Receive and Send Pipeline 

- Pipeline Components

- Adapter and Resolver Framework

- Exception Management

- On Ramp Web Services

- Core Web Services - Resolver, Operations, Exception, UDDI and Transformation Services

Itinerary - An itinerary is a set of steps, where each step performs an action e.g. message transformation, routing, orchestration execution. 

The next three components i.e. Pipelines, Pipeline Components, and Adapter & Resolver Framework together provide dynamic resolution of End Points, Transformation maps and Itinerary based on message context.

Exception Management - provides a framework which can be used not only in BizTalk applications but also in other non BizTalk application to log exceptions at one place.

On Ramp Web Services  - There are number of services exposed once you configure ESB toolkit and primary purpose of these services is allow consumers to submit message to ESB over http protocol. The receive location corresponding to these web services in BizTalk are referred to as On-Ramp. However your On-Ramp Receive locations can also receive messages from other protocol e.g. using File Adapter.

Core Web Services - Each of these services expose specific functionality as follows:

 - Resolver and Transformation - These services extends the dynamic resolution and dynamic transformation functionality to consumer over web service. Behind the scene these work based on Adapter & Resolver Framework only.

- Operations - Operation web service expose BizTalk operation service and has handful of operations to browse various BizTalk artifacts. This is subset of BizTalk OperationOM dll

- Exception - Exception Service allow consumers to save exception to exception store

- UDDI - UDDI service expose UDDI3 feature as web service which can used to search for registered services etc.

How all these components fit together

See the diagram below: It shows various components of ESB and how they interact with each other. Each Marked shape in the diagram is described below:

1. Consumer sends a message to ESB using one of the On-Ramp service. There are 6 On-Ramp services, 3 for one way and 3 for two-way operations. And each operation is supported over ASMX, WCF and WCF Generic. The difference between three different service types is explained in point 3

2. On-Ramp receive location receives the message and executes the pipeline.
3. Depending on which On-Ramp service has been used for submitting the message, There are three ways an itinerary is identified for an incoming message:
a - ASMX Service - Whole itinerary is part of message sent as part of message soap header. In this case consumer sends itinerary xml within message soap header while calling the On-Ramp service. This is does using ESB.ItineraryServices/ProcessItinerary.asmx On-Ramp service for one way operation and ESB.ItineraryServices.Response/ProcessItinerary.asmx for two way operation . In this case while developing itinerary in visual studio, you chose the option of exporting itinerary to xml rather than itineray store
b -  WCF Service - Itinerary is stored in Itinerary store database in ESB environment, and message sender sends the itinerary name, and version along with message. The consumer in this case use ESB.ItineraryServices.WCF/ProcessItinerary.svc On-Ramp service for one way operation and ESB.ItineraryServices.Response.WCF/ProcessItinerary.svc for two way operation.
c - WCF Generic - No itinerary detail is sent along with message in this case, consumer only passes message with not details on how the message will be processed. On ESB side, itinerary selector resolves the itinerary using resolver framework based on message context and load the itinerary from itinerary store for execution. This is rececommanded way, the other two ways are available for backward compatibility with ESB toolkit 1.0. The consumer uses ESB.ItineraryServices.Generic.WCF/ProcessItinerary.svc for one way operation and ESB.ItineraryServices.Generic.Response.WCF/ProcessItinerary.svc for two way operation.

4. Once Itinerary is identified and loaded from message or itinerary store, itinerary selector component  attaches it to the message context. From here onwards, itinerary is part of message context.

5. Based on the steps defined in itinerary, the ESB Dispatcher component of pipeline executes either the Messaging Itinerary service and performs routing/transformation within pipeline component or
6, 7. It publishes the message to Message Box to be picked up by either Orchestration Service or send port (Off-Ramp).

Messaging Itinerary and Orchestration Itinerary Service are two ways you can route/transform messages in ESB dynamically. Messaging Itinerary gets executed within pipeline component scope while orchestration itinerary service is nothing but an execution of an orchestration. So use orchestration service in case you want to execute some business process in addition to routing and transformation. In most other cases Messaging Itinerary Service should suffice.

8. Send Ports (Off-Ramp) also use many ESB pipelines while sending the messages. These sends ports are dynamic send ports, either one-way, two-way, forwarder etc. The three key points to remember while configuring Send Ports and using them in itinerary are:
 - Create them Dynamic
 - Use Filter having minimum following properties
       - Microsoft.Practices.ESB.Schema.ServiceName = Any Name you want to give here, can be any string
       - Microsoft.Practices.ESB.Schema.ServiceType = Messaging, set it to "Messaging" indicating that this port is being used by Messaging Itinerary Service
       - Microsoft.Practices.ESB.Schema.ServiceState = Pending, set it "Pending" always
       - Microsoft.Practices.ESB.Schema.IsRequestResponse = True/False based on whether port is one-way or two-way.
  - Use the appropriate pipeline based on requirement. There are number of pipelines to choose from.

9. Once you configure ESB, Microsoft.Practices.ESB application in BizTalk Admin console contains a send port ALL.Exceptions which consumes all exceptions message from Message Box and writes them to EsbExceptionDb which is used by ESB.Portal application to show faults or generate alerts. The exception can also be written by calling ExceptionService. In your design, ESB Exception Management can become central repository for all exceptions including those which are outside BizTalk environment applications.

10. Adpater Provider and Resolver Framework - In my view, this is the backbone of ESB architecture which is used every where, the best part of it is that it's very extensible.

In simple terms resolver framework helps you resolve the maps name, send port destination details (end points), itinerary name. There are different type of resolver, BRE, STATIC, UDDI etc. The purpose of each resolver is to resolve one of the above three things based on your scenario. e.g. you can store all your endpoint configuration in UDDI and use UDDI resolver to retrieve all that information at run time based on the message context. Consider each different types of resolver as your different configuration store. So each Resolver gets the details based on selected resolver and returns the data in form of a dictionary. That's all it does. 

A resolver is called by either pipeline component like Itinerary Selector, ESB Dispatcher or Messaging/Orchestration Services. After the resolution is done, depending on what is being resolved one of following happens:
a - If its an end point resolution (usually you will do in off Ramp, in routing service), Adapter provider sets adapter properties for dynamic routing. There are different adapter provider for specific adapter, and each of them set required properties of the adapter for dynamic routing.
b - If its for transformation resolution, then transformation service will execute the resolved map as required.
c - If its for itinerary resolution, then itinerary selector component will load the itinerary from itinerary store.

Both Adapter Provider and Resolver can be extended by creating customer adapter provider and resolver by implementing IAdapterProvider and IResolver interface and registering them Esb.Config respectively.

11. ESB Portal is part of ESB Samples with source code. This can be used as a start for managing exceptions within ESB. This allows to generate alert, message correction or message re-submission as well. Also you are free to extend its capability to your need.

12. The core web services as mentioned above can be used by consumers to use ESB feature of dynamic resolution, transformation even in non BizTalk environment. Some of other services like Operations, UDDI gives administrative capability for BizTalk and UDDI as well. The exception service as mentioned above can be used to submit exception to EsbExceptionDb.

I hope it makes it easy to understand ESB Toolkit and its various components.