After upgrading to 1.12.1367 on a tenant you get error "an unsecured or incorrectly secured fault was received"

Problem

Error Message

Message: Could not get next job:
System.ServiceModel.Security.MessageSecurityException:
An unsecured or incorrectly secured fault was received from the other party.
See the inner FaultException for the fault code and detail. ---> System.ServiceModel.FaultException:
At least one security token in the message could not be validated.

   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Solution

This error is caused by mismatch between the a ProvisioningServiceRealm on the tenant and the AudienceURI on the central front ends

Make sure that AudienceUris in web.config on ProcessSystemWeb, frontend-server(s), matches the configured value ProvisioningServiceRealm in ProvisioningSystem.Service.exe.config

On Frontend Servers

  1. Log in to the frontend server in the central environment
  2. Open file C:\Program Files\Zipper\ZervicePoint\Web\ProcessSystem\web.config
  3. Locate audienceUris and note the value located web.config audienceUri Example of audienceUris on the frontend-servers, located in file C:\Program Files\Zipper\ZervicePoint\Web\ProcessSystem\web.config
  4. Repeat for all frontends and make sure that audienceUris are the same for all frontends

On Tenant where error occurs

  1. Log in to the tenant where the problem occurs
  2. Open file C:\Program Files\Zipper\ZervicePoint\ProvisioningSystem\Zipper.ZervicePoint.ProvisioningSystem.Service.exe.config
  3. At ProvisioningServiceRealm located under AppSettings make sure the value matches a value specified on the frontends exe.config appSettings Example of ProvisioningServiceRealm on tenant, located in file C:\Program Files\Zipper\ZervicePoint\ProvisioningSystem\Zipper.ZervicePoint.ProvisioningSystem.Service.exe.config