x9000.com Blog System

History of High Availability Design before Exchange 2010

Prior to the release of Exchange 2007, the high availability and disaster recovery features included with the Exchange Server product were quite limited. Previous versions of Exchange (Exchange 2003 and earlier) could take advantage of Microsoft Cluster Services (MSCS), but this only provided redundancy at the hardware level since the nodes shared the same storage subsystem. If the active cluster node suddenly became unavailable, the Exchange Virtual Server (EVS) and any relevant cluster resources would fail over to the passive node and the end users could then continue their work.

In order to achieve redundancy at the storage level, organizations were forced to invest in replication solutions provided by third-party software vendors and/or storage hardware vendors. Since solutions provided by third party are not supported by Microsoft and typically are quite expensive to implement, the Exchange Product group wanted to provide better high availability and disaster recovery features natively in the Exchange Server product.

Exchange 2007 gave us a whole sleeve of brand new high availability and disaster recovery features such as Local Continuous Replication (LCR), which targeted small organizations and Cluster Continuous Replication (CCR) which targeted medium and large organizations. Later on (with Exchange 2007 SP1) came Standby Continuous Replication (SCR), targeted at organizations of pretty much all sizes. All three features used a new asynchronous replication technology, which worked by shipping log files to a passive copy of a storage group and after inspection replaying these into this passive copy.

Although LCR provided redundancy at the storage level, the feature never really got much attention. The reason behind this was that since the storage group copies had to be stored on a volume local to the Mailbox server, it presented a single point of failure at the hardware level. CCR combined the new asynchronous replication technology introduced by Exchange 2007 with Windows Failover Clustering technology, thereby providing redundancy at the hardware level as well as at the storage level, providing a true high availability solution without any single point of failures.

CCR cluster nodes could be located in separate data centres in order to provide site-level redundancy, but since CCR was not developed with site resiliency in mind, there were too many complexities involved with a multi-site CCR cluster solution

With Exchange 2007 SP1 there came a feature called Standby Continuous Replication (SCR) which made it possible to ship log files to another Exchange 2007 Mailbox Server. Because SCR did not require Windows Failover Clustering, the log files could be shipped from both clustered and non-clustered Mailbox servers (the SCR source) to clustered and non-clustered mailbox servers (SCR target). SCR also introduced the idea of a lag in the replaying of log files of up to seven days, which made it possible to revert back to a copy of the database unaffected by any current issues affecting the active mailbox.

Exchange 2010

With Exchange 2010, we no longer have the concept of Local Continuous Replication (LCR), Single Copy Clusters (SCC), Cluster Continuous Replication (CCR) or Standby Continuous Replication (SCR). Microsoft has now introduced the Database Availability Group (DAG) act as the base component. This means that no matter if you are going to deploy a local or site-level highly available or disaster recoverable solution, you use a DAG. With Exchange 2010, the one and only method to protect mailbox databases is by using DAG.

The primary component in a DAG is a new component called Active Manager. Instead of the Exchange cluster resource DLL (exres.dll) and the associated cluster service resources that were required when clustering Exchange 2007 and previous versions of Exchange, Exchange 2010 now relies on the Active Manager to manage switchovers and failovers (Also called *overs) between mailbox servers part of a DAG. Active Manager runs on all Mailbox servers that are members of a database availability group (DAG). There are two Active Manager Roles: Primary Active Manager (PAM) and Standby Active Manager (SAM). PAM is the Active Manager in a DAG that decides which copies will be active and passive. PAM is responsible for getting topology change notifications and reacting to server failures. The DAG member that holds the PAM role is always the member that currently owns the cluster quorum resource (default cluster group). If the server that owns the cluster quorum resource fails, the PAM role automatically moves to a surviving server that takes ownership of the cluster quorum resource. In addition, if you need to take the server that hosts the cluster quorum resource offline for maintenance or an upgrade, you must first move the PAM to another server in the DAG. The PAM controls all movement of the active designations between a database's copies (only one copy can be active at any specified time, and that copy may be mounted or dismounted). The PAM also performs the functions of the SAM role on the local system (detecting local database and local Information Store failures).

The SAM provides information on which server hosts the active copy of a mailbox database to other components of Exchange that are running an Active Manager client component (for example, RPC Client Access service or Hub Transport server). The SAM detects failures of local databases and the local Information Store. It reacts to failures by asking the PAM to initiate a failover (if the database is replicated). A SAM doesn't determine the target of failover, nor does it update a database's location state in the PAM. It will access the active database copy location state to answer queries for the active copy of the database that it receives.

When a failure affecting the active database occurs, Active Manager uses several sets of selection criteria to determine which database copy should be activated. Active Manager attempts to locate a mailbox database copy that has a status of Healthy, DisconnectedAndHealthy, DisconnectedAndResynchronizing, or SeedingSource, and that meets all of the following criteria:

• It has a content index with a status of Healthy.
• It has a copy queue length less than 10 log files.
• It has a replay queue length of less than 50 log files.

If none of the database copies meet all of the preceding criteria, Active Manager tries to locate a database copy that meets the next set of criteria:

• It has a content index with a status of Crawling.
• It has a copy queue length less than 10 log files.
• It has a replay queue length of less than 50 log files.

If none of the database copies meet all of the preceding criteria, Active Manager tries to locate a database copy that meets the next set of criteria:

• It has a content index with a status of Healthy.
• It has a replay queue length of less than 50 log files.

If none of the database copies meet all of the preceding criteria, Active Manager tries to locate a database copy that meets the next set of criteria:

• It has a content index with a status of Crawling.
• It has a replay queue length of less than 50 log files.

If none of the database copies meet all of the preceding criteria, Active Manager tries to locate a database copy that meets the next set of criteria:

• It has a replay queue length of less than 50 log files.

If none of the database copies meet all of the preceding criteria, Active Manager tries to locate a database copy that meets the next set of criteria:

• It has a content index with a status of Healthy.
• It has a copy queue length less than 10 log files.

If none of the database copies meet all of the preceding criteria, Active Manager tries to locate a database copy that meets the next set of criteria:

• It has a content index with a status of Crawling; and
• It has a copy queue length that is less than 10 log files

If none of the database copies meet all of the preceding criteria, Active Manager tries to locate a database copy that meets the next set of criteria:

• It has a content index with a status of Healthy.

If none of the database copies meet all of the preceding criteria, Active Manager tries to locate a database copy that meets the next set of criteria:

• It has a content index with a status of Crawling.

If none of the database copies meet all of the preceding criteria, Active Manager tries to activate any database copy with a status of Healthy, DisconnectedAndHealthy, DisconnectedAndResynchronizing, or SeedingSource. If it can't find any database copies with this status, it isn't able to automatically activate a database copy.

In each of the preceding passes, if more than one database copy meets all of the preceding criteria, the configured value for ActivationPreference is consulted, and the database with the lowest value is activated and mounted.

Exchange 2010 supports up to 16 servers per DAG, theoretically allowing one copy of a database on each of the servers in the same DAG (i.e. One Active mailbox database and 15 Passive mailbox database copies). The maximum limit of mailbox databases per server is now 100 (up from 50 in Exchange 2007), although the maximum database size is not 2TB, so it is easily possible to just use one mailbox database. As a side note, this has taken the maximum mailbox data size for one DAG to be 16(Number of Servers) x 100(Maximum number of databases) x 2TB (Largest database size = 3200TB or 3.2 Petabytes.

Because of the improvement made with Exchange 2010 DAG, we will now experience much quicker switchovers and failovers between mailbox database copies. They will typically occur in less than 30 seconds, compared to several minutes with CCR in Exchange 2007. In addition, because Outlook MAPI clients now connect to the RPC Client Access service on the Client Access Servers, end users will rarely notice a *-over occurred.

Databases in Exchange 2010 have now been moved to the organizational level instead of the server level where they existed in Exchange 2007 and earlier versions. This also means Exchange 2010 does not have the concept of storage groups any longer. Now there are databases and a log stream associated with each database. One drawback of CCR was that if only one database failed on the active node, a fail-over of all active databases existing on the clustered mailbox server were moved to the passive CCR node. This meant that all users on that had a mailbox stored on the respective mailbox cluster were affected.

Data centre Activation Coordination (DAC) mode is a property setting for a database availability group (DAG). DAC mode is specifically designed for DAGs with three or more members that are extended to two Active Directory sites. DAC mode is disabled by default and should only be enabled for DAGs with three or more DAG members that have been deployed in a multi-data centre configuration. DAC mode shouldn't be enabled for:

• 2 member DAGs where each member is in a different Active Directory site
• 2-16 member DAGs where all members are in the same Active Directory site

DAC mode is used to control the activation behaviour of a DAG when a catastrophic failure occurs that affects the DAG (for example, a complete failure of one of the data centres). When DAC mode isn't enabled, and a failure affecting multiple servers in the DAG occurs, when a majority of servers are restored after the failure, the DAG will restart and attempt to mount databases. In a multi-data centre configuration, this behaviour could cause split brain syndrome, a condition that occurs when all networks fail, and DAG members can't receive heartbeat signals from each other. Split brain syndrome also occurs when network connectivity is severed between the data centres. Split brain syndrome is prevented by always requiring a majority of the DAG members (and in the case of DAGs with an even number of members, the DAG's witness server) to be available and interacting for the DAG to be operational. When a majority of the members are communicating, the DAG is said to have a quorum.

For example, consider a scenario where Liverpool contains two DAG members and the witness server, and the Bath contains two other DAG members. If Liverpool loses power and you activate the DAG in Bath, if the Bath is restored without network connectivity to the Liverpool, the DAG may enter a split brain syndrome.

DAC mode is designed to prevent split brain syndrome from occurring by including a protocol called Data centre Activation Coordination Protocol (DACP). After a catastrophic failure, when the DAG recovers, it won't automatically mount databases even though the DAG has a quorum. Instead DACP is used to determine the current state of the DAG and whether Active Manager should try to mount the databases.

Active Manager stores a bit in memory (either a 0 or a 1) that tells the DAG whether it's allowed to mount local databases that are assigned as active on the server. When a DAG is running in DAC mode (which would be any DAG with three or more members), each time Active Manager starts up the bit is set to 0, meaning it isn't allowed to mount databases. Because it's in DAC mode, the server must try to communicate with all other members of the DAG that it knows to get another DAG member to give it an answer as to whether it can mount local databases that are assigned as active to it. The answer comes in the form of the bit setting for other Active Managers in the DAG. If another server responds that its bit is set to 1, it means servers are allowed to mount databases, so the server starting up sets its bit to 1 and mounts its databases.

But when you recover from a primary data centre power outage where the servers are recovered but WAN connectivity has not been restored, all of the DAG members in the primary data centre will have a DACP bit value of 0; and therefore none of the servers starting back up in the recovered primary data centre will mount databases, because none of them can communicate with a DAG member that has a DACP bit value of 1.

To enable internal Exchange communication with the DAG, an IP address has to be assigned to it. In a situation where the DAG is configured across Active Directory sites, it is necessary to configure several IP addresses with at least one address in each IP Subnet. This can be achieved via DHCP, but in an environment without DHCP, this needs to be assigned using the Exchange Management Shell using the Set-DatabaseAvailabilityGroup command with the DatabaseAvailabilityGroupIpAddresses parameter.

Finally, servers that are DAG members don't use Microsoft Update to install update rollups. Instead, you need to download the update rollups from the Microsoft Download Centre and then install them manually. When you install an update rollup on a server that's a DAG member, several services will be stopped during the installation, including all Exchange services and the Windows Cluster service. The general process for installing update rollups on a DAG member is as follows:

• Suspend activation for the databases on the server being updated.
• Perform a server switchover so that all databases on the server are passive copies. During the switchover process, there will be a brief interruption in service for the mailboxes hosted on the active databases.
• Install the update rollup.
• Resume activation for the databases on the updated server.
• Perform database switchovers as needed.