1. Here's a thread that needs an answer: Advanced Search Question

Error when adding MBX Server to DAG

Discussion in 'Exchange Server Questions' started by Mahmood Jaffer, Aug 19, 2010.

  1. Mahmood Jaffer

    Mahmood Jaffer

    Guest
    Hello,

    I'm running into the following eror when trying to add first member of an Exchange 2010 DAG:

    [PS] C:\Windows\system32>Add-DatabaseAvailabilityGroupServer -Identity DAG1 -MailboxServer SERVER1

    WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file

    " C:\ExchangeSetupLogs\DagTasks\dagtask_2010-08-18_21-43-16.169_add-databaseavailabiltygroupserver.log" .

    A server-side database availability group administrative operation failed. Error: The operation failed with message: A

     error occurred while attempting a cluster operation. Error: Cluster API '" CreateCluster() failed with 0x35. Error: Th

     network path was not found" ' failed. [Server: Server1.domain.corp]

        + CategoryInfo: InvalidArgument: :)) [Add-DatabaseAvailabilityGroupServer], DagTaskOperationFailedException

        + FullyQualifiedErrorId : 67131516,Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer

    AD and DNS are working optimally and i've disabled all OS firewalls and AV software. Any ideas on what might be causing this?

    Thanks

     
  2. Shafaquat Ali

    Shafaquat Ali

    Guest
    Hi,

    The operations get executed in w3wp.exe, which is running as the machine account, and not you.

    So you'll need to open Local Powershell and run:

    add-databaseavailabiltygroupserver labdag1 labmbx03

    and that should work. If it doesn't, please let us know what the new error code is.

    By the way, please try to avoid using the Failover clustering GUI. While you may need to use it with the Beta bits, we're trying hard so that you should never have to do anything with it.

    I hope this will help you.

    Regards.

    Shafaquat Ali.

    M.C.I.T.P Exchange 2007/2010, M.C.I.T.P Windows Server 2008, M.C.T.S OCS Server 2007 R2, URL: http://blog.WhatDoUC.net Phone: +923008210320

     
  3. Milind Naphade

    Milind Naphade

    Guest
    This operation has nothing to do with w3wp.exe first of all. Can you please post the log entries from add-databaseavailabiltygroupserver.log ???

    Additionally, are you trying to do it in a virtualized environment? If yes and you have cloned the disks then you must sysprep the operating system before it can be a part of the DAG.

    Milind Naphade | MCTS:M (Exchange 2007 and 2010) | http://www.msexchangegeek.com

     
  4. Mahmood Jaffer

    Mahmood Jaffer

    Guest
    I'm doing everything through powershell. Here's are the logs from the dagtask log files:

    add-databaseavailabiltygroupserver started on machine SERVER1.

    [2010-08-18T21:27:21] add-dagserver started

    [2010-08-18T21:27:21] commandline: $scriptCmd = {& $wrappedCmd @PSBoundParameters }

    [2010-08-18T21:27:21] Option 'Identity' = 'DAG1'.

    [2010-08-18T21:27:21] Option 'MailboxServer' = 'SERVER1'.

    [2010-08-18T21:27:21] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.

    [2010-08-18T21:27:21] Option 'WhatIf' = ''.

    [2010-08-18T21:27:21] Process: w3wp w3wp.exe:680.

    [2010-08-18T21:27:21] User context = 'NT AUTHORITY\SYSTEM'.

    [2010-08-18T21:27:21] Member of group 'Everyone'.

    [2010-08-18T21:27:21] Member of group 'BUILTIN\Users'.

    [2010-08-18T21:27:21] Member of group 'NT AUTHORITY\SERVICE'.

    [2010-08-18T21:27:21] Member of group 'CONSOLE LOGON'.

    [2010-08-18T21:27:21] Member of group 'NT AUTHORITY\Authenticated Users'.

    [2010-08-18T21:27:21] Member of group 'NT AUTHORITY\This Organization'.

    [2010-08-18T21:27:21] Member of group 'BUILTIN\IIS_IUSRS'.

    [2010-08-18T21:27:21] Member of group 'LOCAL'.

    [2010-08-18T21:27:21] Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.

    [2010-08-18T21:27:21] Member of group 'BUILTIN\Administrators'.

    [2010-08-18T21:27:21] Updated Progress 'Validating the parameters.' 2%.

    [2010-08-18T21:27:21] Working

    [2010-08-18T21:27:21] Mailbox server: value passed in = SERVER1, mailboxServer.Name = SERVER1, mailboxServer.Fqdn = SERVER1.domain.corp

    [2010-08-18T21:27:21] LogClussvcState: clussvc is Stopped on SERVER1.domain.corp. Exception (if any) = none

    [2010-08-18T21:27:21] The IP addresses for the DAG are (blank means DHCP): 192.168.35.100,192.168.2.102

    [2010-08-18T21:27:21] Looking up IP addresses for DAG1.

    [2010-08-18T21:27:24] Failure while trying to resolve DAG1: threw a SocketException: The requested name is valid, but no data of the requested type was found.

    [2010-08-18T21:27:24] Looking up IP addresses for SERVER1.

    [2010-08-18T21:27:24] SERVER1 = [ fe80::105a:66ed:56d1:3456%15, 192.168.6.46, 192.168.2.100 ].

    [2010-08-18T21:27:24] Looking up IP addresses for SERVER1.domain.corp.

    [2010-08-18T21:27:24] SERVER1.domain.corp = [ fe80::105a:66ed:56d1:3456%15, 192.168.6.46, 192.168.2.100 ].

    [2010-08-18T21:27:24] DAG DAG1 has 0 servers:

    [2010-08-18T21:27:24] Updated Progress 'Checking if Mailbox server 'SERVER1' is in a database availability group.' 4%.

    [2010-08-18T21:27:24] Working

    [2010-08-18T21:27:24] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '" OpenCluster(SERVER1.domain.corp) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper" ' failed.. This is OK.

    [2010-08-18T21:27:24] Ignoring previous error, as it is acceptable if the cluster does not exist yet.

    [2010-08-18T21:27:24] DumpClusterTopology: Opening remote cluster DAG1.

    [2010-08-18T21:27:24] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmGetFqdnFailedNotFoundException: An Active Manager operation failed with a transient error. Please retry the operation. Error: A transient error occurred during discovery of the database availability group topology. Error: The fully qualified domain name for node 'DAG1' could not be found.

    at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.<>c__DisplayClass1.<ResolveFqdn>b__0(Object , EventArgs )

    at Microsoft.Exchange.Cluster.ActiveManagerServer.AmHelper.RunADOperationEx(EventHandler ev)

    at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.ResolveFqdn(String shortNodeName)

    at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.GetFqdn(String shortNodeName)

    at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerName..ctor(String serverName)

    at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor(String nameCluster, ITaskOutputHelper output)

    [2010-08-18T21:27:24] Dumping the cluster by connecting to: DAG1.

    [2010-08-18T21:27:24] DumpClusterTopology: Not dumping, since the cluster could not be contacted.

    [2010-08-18T21:27:24] DumpClusterTopology: Opening local cluster.

    [2010-08-18T21:27:24] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '" OpenCluster(SERVER1.domain.corp) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper" ' failed. ---> System.ComponentModel.Win32Exception: There are no more endpoints available from the endpoint mapper

    --- End of inner exception stack trace -
    at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByName(AmServerName serverName)

    at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor(String nameCluster, ITaskOutputHelper output)

    [2010-08-18T21:27:24] Dumping the cluster by connecting to: .

    [2010-08-18T21:27:24] DumpClusterTopology: Not dumping, since the cluster could not be contacted.

    [2010-08-18T21:27:24] The new server (SERVER1) is not a member of a cluster, nor are the other servers (if there are any other servers).

    [2010-08-18T21:27:24] The computer account DAG1 does not exist.

    [2010-08-18T21:27:24] GetNodeClusterState( SERVER1 ) failed with 53. Continuing anyway (assuming clustering is still installed though).

    [2010-08-18T21:27:24] InternalValidate() done.

    [2010-08-18T21:27:24] Updated Progress 'Adding server 'SERVER1' to database availability group 'DAG1'.' 6%.

    [2010-08-18T21:27:24] Working

    [2010-08-18T21:27:24] Connecting to server 'SERVER1.domain.corp' via WMI...

    [2010-08-18T21:27:24] Fetching the network adapters and including the ones without default gateways.

    [2010-08-18T21:27:24] SERVER1.domain.corp has an address: 192.168.6.46/24 default gateway(s)=192.168.6.1 [valid]

    [2010-08-18T21:27:24] IP address derived that's suitable for clustering DHCP/IPv6: 192.168.6.0/24.

    [2010-08-18T21:27:24] SERVER1.domain.corp has an address: fe80::105a:66ed:56d1:3456/64 default gateway(s)=192.168.6.1 [invalid]

    [2010-08-18T21:27:24] SERVER1.domain.corp has an address: 192.168.2.100/24 default gateway(s)=<none> [valid]

    [2010-08-18T21:27:24] IP address derived that's suitable for clustering DHCP/IPv6: 192.168.2.0/24.

    [2010-08-18T21:27:24] Could not find any suitable networks for the address 192.168.35.100!

    [2010-08-18T21:27:24] Ignoring IP address '192.168.35.100'. Can't find a matching network on server SERVER1.

    [2010-08-18T21:27:24] The address 192.168.2.102 falls under the network (192.168.2.0/24).

    [2010-08-18T21:27:24] Updated Progress 'Forming cluster named 'DAG1' on server 'SERVER1'.' 8%.

    [2010-08-18T21:27:24] Working

    [2010-08-18T21:27:24] Forming cluster named 'DAG1' on server 'SERVER1'. The cluster address is/are [192.168.2.102], and the cluster network prefix length is/are [24].

    [2010-08-18T21:27:24] The following log entry comes from a different process, run on machine 'SERVER1'. BEGIN

    [2010-08-18T21:27:24] [2010-08-18T21:27:24] Updated Progress 'Forming cluster named 'DAG1' on server 'SERVER1'.' 2%.

    [2010-08-18T21:27:24] Working

    [2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseInitialize, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 5, szObjectName = DAG1, dwStatus = 0x0 )

    [2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseInitialize, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 5, szObjectName = , dwStatus = 0x0 )

    [2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 10, szObjectName = SERVER1.domain.corp, dwStatus = 0x0 )

    [2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseContinue, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 10, szObjectName = SERVER1.domain.corp, dwStatus = 0x35 )

    [2010-08-18T21:27:24] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 10, szObjectName = SERVER1.domain.corp, dwStatus = 0x35 )

    [2010-08-18T21:27:24] The operation wasn't successful because an error was encountered. You may find more details in log file " C:\ExchangeSetupLogs\DagTasks\dagtask_2010-08-18_21-27-21.899_add-databaseavailabiltygroupserver.log" .

    [2010-08-18T21:27:24] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed with message: An error occurred while attempting a cluster operation. Error: Cluster API '" CreateCluster() failed with 0x35. Error: The network path was not found" ' failed. ---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '" CreateCluster() failed with 0x35. Error: The network path was not found" ' failed. ---> System.ComponentModel.Win32Exception: The network path was not found

    --- End of inner exception stack trace -
    at Microsoft.Exchange.Cluster.ClusApi.AmCluster.CreateExchangeCluster(String clusterName, AmServerName firstNodeName, String[] ipAddress, UInt32[] ipPrefixLength, IClusterSetupProgress setupProgress, IntPtr context, Exception& failureException, Boolean throwExceptionOnFailure)

    at Microsoft.Exchange.Cluster.Replay.DagHelper.CreateDagCluster(String clusterName, AmServerName firstNodeName, String[] ipAddresses, UInt32[] netmasks, String& verboseLog)

    --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.AmClusterApiException) -
    at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)

    at Microsoft.Exchange.Cluster.Replay.DagHelper.CreateDagCluster(String clusterName, AmServerName firstNodeName, String[] ipAddresses, UInt32[] netmasks, String& verboseLog)

    at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass2e.<RpcsCreateCluster>b__2d()

    at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)

    --- End of stack trace on server (SERVER1.domain.corp) -
    at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)

    at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(String serverName, String databaseName, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)

    at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(String serverName, Nullable`1 dbGuid, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)

    at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunCreateCluster(AmServerName rpcServerName, String clusterName, AmServerName firstNode, String[] ipaddrs, UInt32[] netmasks, String& verboseLog)

    at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.FormCluster()

    [2010-08-18T21:27:24] Updated Progress 'Done!' 100%.

    [2010-08-18T21:27:24] COMPLETED

    add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().

     
  5. Scott O'Driscoll

    Scott O'Driscoll

    Guest
    I just had this exact problem. The EX2K10 servers are running Windows 2008 Enterprise R2. If a dedicated replication network has been set up with static routes between each member server, you actually need to specify internal DNS servers that accept dynamic updates for that network interface as the DAG name needs to be registered. I was able to get beyond this step after taking the following action:

    - Stop Cluster Service and set to Disabled for each member server.

    - Add internal DNS servers to DAG interface on each server, and specify DNS suffix.

    - Change binding order to ensure Public/Primary interface is at the top.

    - Unselect all bindings for IPv4/IPv6 stacks on DAG interface.

     
  6. Shafaquat Ali

    Shafaquat Ali

    Guest
    Hi,

    I have seen it fail in adding the server if permissions are not correct for the FSW. Matter of fact it happens in one of the online labs. So if it cannot connect to the FSW share it seems to fail when you try to add the Server to the DAG. this only happens if you bypass the initial warning when setting up the DAG and do not pay attention to the warning's.

    Also check that your Administrative shares are enable for each mailbox server.

    Regards.

    Shafaquat Ali.

    M.C.I.T.P Exchange 2007/2010, M.C.I.T.P Windows Server 2008, M.C.T.S OCS Server 2007 R2, URL: http://blog.WhatDoUC.net Phone: +923008210320

     
  7. prakp

    prakp

    Guest
    From the Logs below it seems to be a network issue.

    2010-08-18T21:27:24] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '" OpenCluster(SERVER1.domain.corp) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper" ' failed. ---> System.ComponentModel.Win32Exception: There are no more endpoints available from the endpoint mapper

    How are the nodes of DAG connected to each other? VPN..etc?

    Have you tried adding the nodes manually using Failover Cluster Manager?

     
  8. Milind Naphade

    Milind Naphade

    Guest
    You go the point Prakash :) (I hope I am not making any mistake)

    Okay, so error code 0x6d9 translates into EPT_S_NOT_REGISTERED and error code 0x35 into ERROR_BAD_NETPATH. According to this error it is a RPC endpoint unavailability problem which may also be a biproduct of network failures between DAG nodes. In this case I would run a netdiag on the servers in question and see if it returns any errors.

    Make sure the server where you have specified your FSW (HT server in your case as you have not specified any specific server) is accessible from all DAG members. Can you try comparing SIDs of both servers with psgetsid.exe from sysinternals if possible?

    Milind Naphade | MCTS:M (Exchange 2007 and 2010) | http://www.msexchangegeek.com

     
  9. James-Luo

    James-Luo

    Guest
    In your log, I didn"t see the task for the Windows Failover Clustering component installation. Did you install it before on the SERVER1?

    Please check the settings of the NIC, incorrect configuration will cause the similar issue

    &ldquo;Network Adapter Configuration for DAGs&rdquo; section in Planning for High Availability and Site Resilience

    James Luo

    TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx)

    If you have any feedback on our support, please contact tngfb@microsoft.com

     
  10. James-Luo

    James-Luo

    Guest
  11. prakp

    prakp

    Guest
    i guess i know whats going on with this issue:

    If you have a Site To Site VPN configured check the Rule if its working as its suppose to. This is what i had to do in one of the instance

    Checkpoint R71, NAT within VPN communities should work by default, however had to push it manually. For that we will have to create a RULE/Configuration and push it through CheckPoint.

    Let me know if it helps!!

     

Share This Page