Software Load Balancer and Distributor

The purpose of this module is to manage a collection of service endpoints (web service address) in a way that optimizes the connection performance and or load distribution across endpoints.

Scenario #1:
We have two endpoints: http://myservice1 /service.svc and http://myservice2 /service.svc. They provide the same functionality but are hosted in two different servers for reliability. For this scenario, we want to be able to connect to either endpoint making sure that the one we pick is the one that is currently having the least load.

Scenario #2:
We have a set of endpoints and we want to affinitize the requests to specific endpoints based on some value. The rule that determines how the endpoint is chose varies by application. Once could say for example, all customers that start with ‘A’ use this endpoint, etc.

The ability to affinitize request to endpoint is very useful to enable partitioning scenarios but many complications arise, in particular when new endpoints are added or when endpoints fail.

Scenario #3:
Similar to #2 but we want to first select a specific cluster of endpoints based on some value and then the endpoint in that cluster. For example you can have n identical clusters, each one exposing a collection of endpoints. Imagine a commerce service where several clusters host data for different group of customers.

The module can run in two modes local or service. Local mode is for very simple scenarios or for developer/test environments where hosting the service might be cumbersome.

In most scenarios you will run a service and a client component. The service uses a simple database to store configuration parameters and status (current load, failures, logs). In local mode the configuration is stored in an XML file.

The configuration consists of Applications, Logical Endpoints and Physical Endpoints.

Applications are clusters such as “storage service”, “commerce”, etc.

Physical Endpoints are the URL’s of the application services.

Logical Endpoints are virtual endpoints and are important only when using any type of affinity/partitioning.

The assumption is that the number of logical endpoints will remain constant so care must be taken during initial configuration to select the right number of logical endpoints. Even if starting with a low number of physical endpoints, let say three, you can pick 1000 logical endpoints. These 1000 endpoints will initially connect to one of the three endpoints. As physical endpoints are added (or fail) the mapping from logical to physical will change but the application will not be affected and the load is automatically redistributed.

 // Request a load balancer object for my application
LoadBalancer lb = LoadBalancer.Get( “myApplicationName”);

// Calculate a partition key based on some value
int partition = CustomerNumber  mod lb.Endpoints.Count;

// Get an endpoint physical address
EndPoint ep = lb.GeEndpoint(partition);

MyServiceProxy client = new MyServiceProxy();

do
{

Client.endpoint = ep;

// open the connection.
Client.open();

….

If failure then do

                ep = Lb.Failover(ep);  

…….

} while (failure…)

//close and update the endpoint status and load
Client.close()

The service stores the application configuration and also tracks the status of the different endpoints. Clients sent asynchronously information about the load and response times they encounter when taking to endpoints. The service stores this information and uses it to rank endpoints and select the best replacement endpoint in the event of a failure.

The service implements a failover method that allows a client to indicate that an endpoint has failed and a new one needs to be assigned. Logic on the service side prevents reassigning endpoints when multiple clients call to report a failover for the same endpoint.

There are also management services to retrieve the status of endpoints and to manually remove/add endpoints to an application at runtime.