Monday, November 14, 2011

Exchange 2010 DAG transaction logs are not truncated after VSS Full backup

A couple of months ago we implemented for a customer a hosting solution based on Exchange 2010. The solution has full redundancy for all components including a DAG with geographical redundancy.
Everything was configured accordingly to Microsoft recommendations and worked without any problems.

We configured Windows Server Backup to perform backups using a DAG member that was hosting only active databases. This means that we didn't have to do the Registry modifications as mentioned in TechNet (http://technet.microsoft.com/en-us/library/dd876851.aspx).
The backup was successful but not logs were deleted from the log's folder. In the Event Viewer the only relevant record was:

Information Store (2672) DB1: No log files can be truncated.

Exchange VSS Writer (instance acedea2e-c785-3e33-8257-ac38405e81af:5) has successfully completed the full or incremental backup of replicated database 'DB1'. The log files will be truncated after they have been replayed.

We tried to find a solution and nothing worked and we had to give up for a while in using DAG replication. Microsoft support has extensively investigated the issue for more than two months without success. In the end their suggestion was that between the geographically dispersed members of the DAG there was something that was blocking the traffic.

We discussed with the network infrastructure admin and all the ports seems to be open. The only thing that could block the communication was a Network Intrusion Detection System. This system was blocking SUNRPC and DCERPC protocols. After we disabled this protection and run the backup, all database logs were truncated.

If you encounter this issue, try to disable everything that could potentially block the communications between DAG members.

Sunday, November 13, 2011

Unable to authenticate users in Office 365 using ADFS services

Last week we migrated one of our clients to Office 365 including the configuration of ADFS services in order to authenticate the users using local Active Directory credentials. Everything worked as expected for a couple of days until one morning when nobody could authenticate to the service.
We investigated the ADFS server and we found out that the AD FS 2.0 Windows Service was stopped.
We tried to start it without success. At a deeper inspection we discovered that the account that should be used for running this account was deleted from AD.
We created a new account, assigned all the appropriate rights and we managed to start the service. At this point, the external users that were authenticated using the ADFS Proxy server were able to login to the Office 365 services but the internal users were unable to do so.
In the AD FS 2.0/Admin event viewer log there were a lot of events like:

Log Name: AD FS 2.0/AdminSource: AD FS 2.0
Date:          11/8/2011 10:15:12 AM
Event ID: 323Task Category: None
Level: ErrorKeywords: AD FS
User: DOMAIN\ServiceAccount
Computer:     COMPUTER.domain.com
Description:
The Federation Service could not authorize token issuance for the caller 'DOMAIN\ServiceAccount' on behalf of the subject 'user@domain.com
' to the relying party 'urn:federation:MicrosoftOnline'
. Please see event 501 with the same instance id for caller identity. Please see event 502 with the same instance id for OnBehalfOf identity, if any.
Additional Data
Instance id: 2ae34f5b-dc27-3ed1-af21-976e28a4e3f5
Exception details:
Microsoft.IdentityServer.Service.IssuancePipeline.OnBehalfOfAuthorizationException: MSIS5009: The impersonation authorization failed for caller identity and delegate for relying party trust https://claimapp1.treyresearch.net. at Microsoft.IdentityModel.Threading.AsyncResult.End(IAsyncResult result)
at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceContract.ProcessCoreAsyncResult.End(IAsyncResult ar)
at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceContract.EndProcessCore(IAsyncResult ar, String requestAction, String responseAction, String trustNamespace)
User Action
Use Windows PowerShell comments for AD FS 2.0 to ensure that the caller is authorized on behalf of the subject to the relying party.

Log Name: AD FS 2.0/AdminSource: AD FS 2.0
Date:          11/8/2011 10:11:02 AM
Event ID: 364Task Category: None
Level: ErrorKeywords: AD FS
User: DOMAIN\ServiceAccount
Computer: COMPUTER.domain.com
Description:
Encountered error during federation passive request.
Additional Data
Exception details:
Microsoft.IdentityServer.Web.RequestFailedException: MSIS7012: An error occurred while processing the request. Contact your administrator for details. ---> System.ServiceModel.FaultException: MSIS3126: Access denied. at Microsoft.IdentityServer.Protocols.WSTrust.WSTrustClientManager.Issue(Message request, WCFResponseData responseData)
at Microsoft.IdentityServer.Protocols.WSTrust.WSTrustClient.Issue(RequestSecurityToken rst, WCFResponseData responseData)
at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.SubmitRequest(MSISRequestSecurityToken request)
--- End of inner exception stack trace ---
at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.SubmitRequest(MSISRequestSecurityToken request)
at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.RequestBearerToken(MSISSignInRequestMessage signInRequest, SecurityTokenElement onBehalfOf, SecurityToken primaryAuthToken, String desiredTokenType, Uri& replyTo)
at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.RequestBearerToken(MSISSignInRequestMessage signInRequest, SecurityTokenElement onBehalfOf, SecurityToken primaryAuthToken, String desiredTokenType, MSISSession& session)
at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.BuildSignInResponseCoreWithSerializedToken(String signOnToken, WSFederationMessage incomingMessage)
at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.BuildSignInResponseForProtocolResponse(FederationPassiveContext federationPassiveContext)
at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.BuildSignInResponse(SecurityToken securityToken)
System.ServiceModel.FaultException: MSIS3126: Access denied. at Microsoft.IdentityServer.Protocols.WSTrust.WSTrustClientManager.Issue(Message request, WCFResponseData responseData)
at Microsoft.IdentityServer.Protocols.WSTrust.WSTrustClient.Issue(RequestSecurityToken rst, WCFResponseData responseData)
at Microsoft.IdentityServer.Web.FederationPassiveAuthentication.SubmitRequest(MSISRequestSecurityToken request)

There were also a lot of events 501 and 502.
We searched the internet for a solution and all we've found was http://social.technet.microsoft.com/wiki/contents/articles/2265.aspx which unfortunately did not work.
The only possible solution that came out was to disable the federation between Office 365 and internal AD infrastructure and re-enable it using a new service account for the AD FS service. To do that we had to follow the procedure:

1. Use PowerShell command Convert-MsolDomainToStandard to convert specified domain from single sign-on  to standard authentication.

If you didn't install the Office 365 cmdlets please follow the instructions from this blog post http://paulroman.pras.ro/2011/11/unable-to-sign-in-to-office-365-with-ad.html.

Run the following commands to disable single sign-on:

$cred = Get-Credential
Connect-MsolService -cred $cred         
convert-MsolDomainToStandard –DomainName domain.com –passwordfile password.txt –SkipUserConversion $false


This will generate temporary passwords for all the users and save them to the password.txt file.

2. Uninstall all ADFS and ADFS Proxy services from all the servers that were members in this process.

3. Following the Microsoft Procedure from http://onlinehelp.microsoft.com/en-us/office365-enterprises/hh125004.aspx reconfigure the ADFS integration with Office 365.

In no more than half an hour the ADFS service was functional again and all the users were able to access Office 365 from internal or external network.

Saturday, November 12, 2011

Unable to sign in to Office 365 with AD federated user

Last week, one of our customer's employee had trouble signing in to his Office 365 services using his federated account. All other accounts were working fine except this one. After investigating the issue we found out that the account UPN was changed because the user changed her name after she got married.
We created an incident in the Office 365 admin console and they suggested us a solution that did not work. After a couple of hours spent searching online for solutions we've created the following procedure.

This procedure should be used anytime you want to change the UPN for a federated user.

Connect to Office 365 using Powershell

Install the Office 365 cmdlets

To begin using the Office 365 cmdlets, you first need to install them. The requirements for installing the Office 365 cmdlets are as follows:
• You can install the cmdlets on a Windows 7 or Windows Server 2008 R2 computer.
• You must have Windows PowerShell and the .NET Framework 3.5.1 installed.
• You must have the Microsoft Online Services Sign-in Assistant installed. For more information, see Manually update and configure desktops for Office 365.

To install the cmdlets, perform the following steps.
1. Download one of the following:
Microsoft Online Services Module for Windows PowerShell (32-bit version)
Microsoft Online Services Module for Windows PowerShell (64-bit version)
2. To install the cmdlets, double-click the AdministrationConfig.msi file.
The installer will add a shortcut to your desktop and Start menu. Click the Microsoft Online Services Module shortcut to open a Windows PowerShell workspace with the cmdlets.
Alternatively, you can also load the Office 365 cmdlets manually by typing import-module MSOnline at the Windows PowerShell prompt.

Change the UPN suffix and disable single sign on for that user

Run from powershell the following command to disable temporary the single sign on and to change the UPN for that user:
set-msoluserprincipalname – UserPrincipalName username@domain.com -new UserPrincipalName username@domain.onmicorsoft.com

The command will allocate the UPN username@domain.onmicrosoft.com to the user and it will generate a temporary password. The password can be used to login to Office 365

Change the UPN for the user

Use Active Directory Users and Computers MMC to change the UPN for the user and force the synchronization between AD Office 365 by running DirSyncConfigShell.psc1 from C:\Program Files\Microsoft Online Directory Sync on the server that has the DirSync tool installed.

Run the following command:

Start-OnlineCoexistenceSync

This will change automatically the UPN for the user to the one that is defined in AD and we re-enable the single sign on process for that user.

Delete ADFS cache from the ADFS servers (not ADFS proxy)

To work around this issue, disable the local SID cache on the domain member computer. To do this, follow these steps:
1. Open Registry Editor.
2. Locate and then right-click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
3. Point to New, and then click DWORD Value.
4. Type LsaLookupCacheMaxSize, and then press ENTER.
5. Right-click LsaLookupCacheMaxSize, and then click Modify.
6. In the Value data box, type 0, and then click OK.
7. Exit Registry Editor.
Note The LsaLookupCacheMaxSize registry entry sets the maximum number of cached mappings that can be saved in the local SID cache. The default maximum number is 128. When the LsaLookupCacheMaxSize registry entry is set to 0, the local SID cache is disabled.

Verify that the user can successfully login in into Office 365 using his federated account and delete the previous registry key.