Thursday, December 1, 2011

[OpsMgr 2007] Timeout running Remove-DisabledMonitoringObject cmdlet in System Center Operations Manager

I've read a lot and seen a lot of cases like mine - I mean remove-disabledmonitoringobject cmdlet get the error "The requested operation timed out" but it seems that nobody has been lucky to found how to fix the issue. We have openned a Micorsoft case on this issue a lot of monthes ago and since the begining of the week, we are now able to run the cmdlet without any error.

This resolution given by Microsoft for an openned case will not be part of the CU6.
The resolution has been found too late !


First, here is our configuration - approximatly 3000 agents in CU3. We don't have upgraded to CU4 and we soon upgrade to CU5. SQL 2008 for the DBs.

The first thinking of why the cmdlet is timing out was the number of overrides was too much for the SDK to process before the thirty minute WCF(Windows Connection Framework) timeout occurs.

Within our production environment there are approximately 140000 DiscoverySources which need analyse be the cmdlet to know if the associated discovered types need to be removed or not. We have a pre-production environnemnt with less number of agents and only 40000 DiscoverySources on wich the cmdlet is well working.

To reduce the number of Discovery sources we have analysed all the MP we have and it appeared that OCS MP was responsible for almost 40% of the hugh number of DiscoverySources. The OCS MP has nineteen discoveries targeted at Windows Server Computer class enabled by default. On each discoveries we have an override on a group to disable disovery for the group members. A discoverysource entry is created for each discovery-to-target-entity mapping.
We have also :  19 * ~3000 agents = 57000 discoverysources just for OCS MP.
When overrides are done one theses discoveries, the enabled states must calculated for all discoverysources !

I've worked to remove some overrides and also to reduce the enabled state calculation for the DiscoverySources but the cmdlet was always timed out.

The next way to fix the issue was to let Microsoft have an other review of the code and SQL involved to see if they can make some efficiencies in the way they do this. I've also been asked to run 2 queries on the SCOM Database :
I’ve also run the following queries :

  1. SELECT COUNT (Distinct [DiscoverySource].[DiscoverySourceId])
  2. FROM dbo.DiscoverySource
  3. INNER JOIN dbo.ModuleOverride ON ModuleOverride.ParentId = DiscoverySource.DiscoveryRuleId
  4. AND ModuleOverride.OverrideableParameterId = dbo.fn_MPObjectId(NULL, NULL, N'Enabled')
  5. AND (ParentType = 'Discovery' OR ParentType = 'Rule')
  6. join DiscoverySourceToTypedManagedEntity dstme
  7. on discoverysource.DiscoverySourceId = dstme.DiscoverySourceId
  8. WHERE DiscoverySource.IsDeleted = 0
  9. AND ModuleOverride.Value = 'false'

  1. SELECT COUNT (Distinct [DiscoverySource].[DiscoverySourceId])
  2. FROM dbo.DiscoverySource
  3. INNER JOIN dbo.ModuleOverride ON ModuleOverride.ParentId = DiscoverySource.DiscoveryRuleId
  4. AND ModuleOverride.OverrideableParameterId = dbo.fn_MPObjectId(NULL, NULL, N'Enabled')
  5. AND (ParentType = 'Discovery' OR ParentType = 'Rule')
  6. join DiscoverySourceToTypedManagedEntity dstme
  7. on discoverysource.DiscoverySourceId = dstme.DiscoverySourceId
  8. WHERE DiscoverySource.IsDeleted = 0
Given the numbers returned by the queries Microsoft support suspects the step below will allow the cmdlet to complete and await your results.
Here is also what I've been asked to do :
  • Run the SQL against the OperationsManager DB.

  1. DECLARE @querydef XML
  2. SET @querydef =
  3. N'<QueryDefinitions xmlns="urn:DataAccess" xmlns:dal="urn:DataAccess" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:DataAccess QueryDefinition.xsd">
  4. <QueryDefinition>
  5.   <Name>DiscoverySourcesEligibleForDeletionDueToOverrides</Name>
  6.   <ObjectName>DiscoverySourcesEligibleForDeletionDueToOverrides</ObjectName>
  7.   <UsedBy Component="Sdk" />
  8.   <Description>Selects discovery sources that *may* be invalid due to applied overrides.</Description>
  9.   <DataObject xsi:type="SelectType">
  10.     <Column>
  11.       <Name>DiscoverySourceId</Name>
  12.       <Source>DiscoverySource</Source>
  13.       <Type>uniqueidentifier</Type>
  14.     </Column>
  15.     <Column>
  16.       <Name>DiscoverySourceType</Name>
  17.       <Source>DiscoverySource</Source>
  18.       <Type>tinyint</Type>
  19.       <EnumType>Microsoft.EnterpriseManagement.Mom.Modules.DataItems.Discovery.DiscoverySourceType</EnumType>
  20.       <EnumLeastValue>Rule</EnumLeastValue>
  21.       <EnumGreatestValue>ConfigService</EnumGreatestValue>
  22.     </Column>
  23.     <Column>
  24.       <Name>DiscoveryRuleId</Name>
  25.       <Source>DiscoverySource</Source>
  26.       <Type>uniqueidentifier</Type>
  27.     </Column>
  28.     <Column>
  29.       <Name>BoundManagedEntityId</Name>
  30.       <Source>DiscoverySource</Source>
  31.       <Type>uniqueidentifier</Type>
  32.     </Column>
  33.     <Argument>Distinct</Argument>
  34.     <Source>
  35.       <Table>
  36.         <Name>DiscoverySource</Name>
  37.         <Owner>dbo</Owner>
  38.         <Type>Table</Type>
  39.       </Table>
  40.       <Join>
  41.         <Type>Inner</Type>
  42.         <Table>
  43.           <Name>ModuleOverride</Name>
  44.           <Owner>dbo</Owner>
  45.           <Type>Table</Type>
  46.         </Table>
  47.         <JoinCondition>ModuleOverride.ParentId = DiscoverySource.DiscoveryRuleId AND ModuleOverride.OverrideableParameterId = dbo.fn_MPObjectId(NULL, NULL, N''Enabled'') AND (ParentType = ''Discovery'' OR ParentType = ''Rule'')</JoinCondition>
  48.       </Join>
  49.       <Join>
  50.         <Type>Inner</Type>
  51.         <Table>
  52.           <Name>DiscoverySourceToTypedManagedEntity</Name>
  53.           <Owner>dbo</Owner>
  54.           <Type>Table</Type>
  55.         </Table>
  56.         <JoinCondition> discoverysource.DiscoverySourceId = DiscoverySourceToTypedManagedEntity.DiscoverySourceId</JoinCondition>
  57.       </Join>
  58.     </Source>
  59.     <Conditional>
  60.       <Condition>
  61.         <Expression>DiscoverySource.IsDeleted = 0 AND ModuleOverride.Value = ''false''</Expression>
  62.       </Condition>
  63.     </Conditional>
  64.   </DataObject>
  65. </QueryDefinition>
  66. </QueryDefinitions>'
  67. INSERT INTO dbo.[DataAccessLayerSetting]([SettingType], [SettingData]) VALUES (0, @querydef)



The impact of this change is when the cmdlet is run, it will now use the SQL query inserted in the Data.AccessLayerSetting table instead of the Original SQL query which is compiled into one dll. This table allows us to override the in-built queries, as it is read on OMSDK service restart.Coming back to the origicnal SQL is very easy, just remove the entry in the Data.AccessLayerSetting table and restart the SDK.

  • Restart the OpsMgr SDK service.
  • Run the remove-disabledmonitoringobject cmdlet and report back on the success or failure of it.
I use to launch the cmdlet like this (in a short PS1):
  1. get-managementserver | select ManagementGroup -unique
  2. get-date
  3. remove-disabledmonitoringobject
  4. get-date
That permit to show in the same few line the OpsMgr group, the dates before and after the cmdlet has run.


 Unfortunatly the 2 first time I've launched the remove-disabledmonitoringobject cmdlet, it went to a new error :

  1. >get-date
  2. Monday, November 28, 2011 8:27:16 AM
  3. PS Monitoring:\
  4. >remove-disabledmonitoringobject
  5. Remove-DisabledMonitoringObject : Microsoft.EnterpriseManagement.Common.DiscoveryDataFromRuleTargetedToDeletedMonitoringObjectException: Discovery data has been received from a rule targeted at a non-existent monitoring object id.
  6. MonitoringObjectId: c1537246-ec53-cc7c-b45f-aed87f06bc7f
  7. RuleId: 66b6d462-535f-cab6-eb14-b24fc79dfb75
  8.    at Microsoft.EnterpriseManagement.DataAbstractionLayer.InstanceSpaceOperations.DeleteDisabledDiscoverySources()
  9.    at Microsoft.EnterpriseManagement.ManagementGroup.DeleteDisabledMonitoringObjects()
  10.    at Microsoft.EnterpriseManagement.OperationsManager.ClientShell.RemoveDisabledMonitoringObjectCmdlet.ProcessRecord()
  11. At line:1 char:32
  12. + remove-disabledmonitoringobject <<<<
  13.     + CategoryInfo          : InvalidOperation: (Microsoft.Enter...ingObjectCmdlet   :RemoveDisabledMonitoringObjectCmdlet) [Remove-DisabledMonitoringObject], Disc
  14.   overyDataFr...ObjectException    + FullyQualifiedErrorId : ExecutionError,Microsoft.EnterpriseManagement.Operat
  15.    ionsManager.ClientShell.RemoveDisabledMonitoringObjectCmdlet
  16. PS Monitoring:\
  17. >get-date
  18. Monday, November 28, 2011 8:48:55 AM
I've also report this again to the microsoft support and wait any answer. That particular seems to be known and the cmdlet completed without error on the second run. In my case, the second run of the cmdlet found a different object which caused again the error and as in the first run it set the original object’s IsDeleted property to 1. It appears that the cmdlet attempts remove some objects twice, and it's failing on the second attempt.
I've launched a third time the cmdlet and it ended in success ! 





This posting is provided "AS IS" with no warranties.

This posting is provided "AS IS" with no warranties.

[SCOM 2007] Authoring Management Pack - Create a VBS discovery with a debugging functionnality - PART II

Now we have all needed function (see PART I of this article), we can create a new Management Pack by using the authoring console.

Create a new MP with the Authoring Console : MyNewMP.xml witha "Microsoft.Windows.Local Application" Class nammed MyNewMP.CLS1.
Add a new property to this class : DebugMod



Then create a new discovery MyNewMP.DSC1 that target Microsoft.Windows.Server.Computer and discover the class MyNewMP.CLS1 and all its attributes.




Choose the disovery type : Microsoft.Windows.TimedScript.DiscoveryProvider and schedule it once a day and call your script MyNewMP.DSC1.script.vbs. Click on Parameters and add 3 parameters like :

$MPElement$ $Target/Id$ $Target/Property[Type="Windows!Microsoft.Windows.Computer"]/PrincipalName$

In the Script field you will have to paste the discovery script below :

  1. First part of the discovery script is to declare and set the variable
  2. Option Explicit
  3. SetLocale("en-us")
  4. '-----------------------------------------------------------------------------------------------------
  5. ' DEFAULT VARIABLES AND CONST - Used by the debuging functions and sub
  6. '-----------------------------------------------------------------------------------------------------
  7. Const REGKEYPATH = "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\Debugging"
  8. Const REGISTRYDEFVALUE = "FALSE"
  9. Const MOMEVENTLOGERROR = "1"
  10. Const MOMEVENTLOGWARNING = "2"
  11. Const MOMEVENTLOGINFORMATION = "4"
  12. Dim intIndex
  13. Dim strRegValue
  14. Dim strRegKeyName
  15. Dim strMPScriptName
  16. Dim strCurrentDebugModeValue
  17. Dim oAPI
  18. Dim oInst
  19. Dim oArgs
  20. Dim oDiscoveryData
  21. '-----------------------------------------------------------------------------------------------------
  22. ' SET THE FOLLOWING VALUES ACCORDING TO YOUR MANAGEMENT PACK
  23. '----------------------------------------------------------------------------------------------------
  24. strMPScriptName = "MyNewMP.DSC1.vbs"
  25. strRegKeyName = "Verbose.MyNewMP"
  26. '-----------------------------------------------------------------------------------------------------
  27. ' Create the object that connect to MOM API
  28. Set oAPI = CreateObject("MOM.ScriptAPI")
The second part of the discovery will be dedicated to all function seens in Part I of this article :

  1. '-----------------------------------------------------------------------------------------------------
  2. ' DEFAULT FUNCTIONS - Used for Management Pack debugging
  3. '-----------------------------------------------------------------------------------------------------
  4. Private Function fct_ReadRegistryKey(ByVal strKeyPath, ByVal strKeyName)
  5. 'Read Registry key located on KeyPath\KeyName
  6. 'Returns the value of a registry key and eventually a no-value to identify that the key does not exists.
  7.            Dim oWshShell
  8.             Dim strRegReadValue     'Contains the value of the Registry Key
  9.             'Try to read the registry key located to strKeyPath\strKeyName
  10.             On Error Resume Next
  11.             Set oWshShell = CreateObject("Wscript.Shell")
  12.             strRegReadValue = oWshShell.RegRead(strKeyPath &"\" & strKeyName)
  13.             'If an error is raised Then we a no-value to identify that the Registry Key does not exist.
  14.             If err.number <> 0 Then
  15.                         strRegReadValue = ""
  16.             End if
  17.             On error goto 0
  18.             fct_ReadRegistryKey = UCase(strRegReadValue)
  19. End Function
  20. '-----------------------------------------------------------------------------------------------------
  21. Private Function fct_WriteRegistryKey(ByVal strKeyPath, ByVal strKeyName, ByVal strKeyValue)
  22. ' Write a Registry value to strKeyPath\strKeyName with value strKeyValue
  23. ' blnKeyEdit is set to True or false in case of error.
  24.             Dim blnKeyEdit
  25.             Dim oWshShell
  26.             blnKeyEdit = false
  27.             'Try to set strKeyValue to the registry key strKeyPath\strKeyName
  28.             On Error Resume Next
  29.             Set oWshShell = WScript.CreateObject("WScript.Shell")
  30.             oWshShell.RegWrite strKeyPath & "\" & strKeyName , UCase(strKeyValue)
  31.             'If setting the registry key strKeyValue is OK Then the function will return a string TRUE.
  32.             If err.number = 0 Then
  33.                         blnKeyEdit = true
  34.             End if
  35.             On error goto 0
  36.             fct_WriteRegistryKey = blnKeyEdit
  37. End Function
  38. '-----------------------------------------------------------------------------------------------------
  39. Private Sub sub_LogMPScriptEvent(ByVal intErrNumber, ByVal strEventLogMessage, ByRef objError, ByVal strDebugMod, ByVal strScriptName, ByVal strMomEventLevel)
  40. ' Log an event in the OperationManager eventLog of the server if it is in debug mode
  41. ' It will allow to trace the debugging process of the script of the Management Pack by logging errors and parameters values
  42.             Dim strMessage
  43.              strMessage=""
  44.              'If the debug mode is ON, an event is logged in the Operation Manager eventLog.
  45.             If ucase(strDebugMod) = "TRUE" Then
  46.                         strMessage = strEventLogMessage & vbCrLf & " " & vbCrLf & _
  47.                         "Error number:" & vbTab & CStr(objError.Number) & vbCrLf & _ 
  48.                         "Error description:" & vbTab & objError.Description
  49.             Call oAPI.LogScriptEvent(strScriptName, intErrNumber, strMomEventLevel, strMessage)
  50.             End if
  51. End Sub
  52. '-----------------------------------------------------------------------------------------------------
  53. Private function fct_LogRunningAccount(ByVal strDebugMode)
  54. ' Log an event in the OperationManager eventLog of the server if it is in debug mode
  55. ' It allows to know under which credentials the script is executed
  56.             Dim owshNetwork
  57.             Dim strRunningUserID
  58.             On Error Resume Next
  59.             Set owshNetwork = WScript.CreateObject("WScript.Network")
  60.             'Assign user name returned to a variable
  61.             strRunningUserID = owshNetwork.UserName
  62.             
  63.             call sub_LogMPScriptEvent (4005, "sub_LogMPScriptEvent. Script running as user: " & strRunningUserID , Err, strDebugMode,strMPScriptName,MOMEVENTLOGINFORMATION)
  64.             On error goto 0
  65.             fct_LogRunningAccount = strRunningUserID
  66. End Function
  67. '-----------------------------------------------------------------------------------------------------
  68. private sub sub_LogScriptStartInfo(byRef oArgs, byVal strDebugModeValue, byval strMomLevelEvent, byVal strScriptName)
  69. ' Log an event with the default informations. The event contains :
  70. ' - The account used to run the script
  71. ' - The number of script parameters
  72. ' - The values of the parameters
  73. ' - The value of the debug mode
  74.             Dim owshNetwork
  75.             Dim strLogMessage
  76.             Dim strRunAsAccount
  77.             On Error Resume Next
  78.             Set owshNetwork = WScript.CreateObject("WScript.Network")
  79.             'Assign user name returned to a variable
  80.             strRunAsAccount = owshNetwork.UserName
  81.              strLogMessage=""
  82.             ' Create the start log Script with launch information data RunAs account, debugmode, arguments)
  83.             strLogMessage = vbcrlf & "The script has been launched under following credentials : " &             strRunAsAccount & "." & vbcrlf & vbcrlf & oArgs.count & " arguments have been passed in parameter : "
  84.             For intIndex = 0 To oArgs.Count-1
  85.                         strLogMessage = strLogMessage & vbcrlf & " - " & oArgs(intIndex)
  86.             Next
  87.             strLogMessage = strLogMessage & vbcrlf & vbcrlf & "The debug mode value for the Management Pack is set to : " & strDebugModeValue
  88.             Call oAPI.LogScriptEvent(strScriptName, 10212, strMomLevelEvent, strLogMessage)
  89. End sub
  90. '-----------------------------------------------------------------------------------------------------
 And the next part of the discovery will be the main part that will test the number of arguments and create / read the debugging registry key :
  1. '-----------------------------------------------------------------------------------------------------
  2. ' MAIN CODE START HERE
  3. '-----------------------------------------------------------------------------------------------------
  4. ' Gets the arguments passed in parameters of the script
  5. Set oArgs = Wscript.Arguments
  6. ' If the minimals arguments are not set then we exit the script
  7. if oArgs.Count < 3 Then
  8.    Wscript.Quit -1
  9. End If
  10. ' Check the value of the registry key on the server to know if the debug mode is ON or not
strRegValue = ""
  1. 'By default the registry key is set to : DOES NOT EXIST. See fct_ReadRegistryKey for more information
  2. strRegValue = fct_ReadRegistryKey(REGKEYPATH, strRegKeyName)
  3. 'If the registry key is TRUE XOR FALSE then the debugmodevalue has a good value
  4. If ((strRegValue = "TRUE") Xor (strRegValue = "FALSE")) Then
  5.             strCurrentDebugModeValue = strRegValue
  6. Else
  7.             'If the registry key is empty or not correctly set then a attempt to set to default value is made.
  8.             If fct_WriteRegistryKey(REGKEYPATH, strRegKeyName, REGISTRYDEFVALUE) Then
  9.                         ' write event 10211 that says the registry key has been created
  10.                         call sub_LogMPScriptEvent (10211, "Registry Key " & REGKEYPATH & "\" & strRegKeyName & " With Value = " & REGISTRYDEFVALUE & " has been created", Err, True,strMPScriptName,MOMEVENTLOGINFORMATION)
  11.             Else
  12.                         ' write event 10210 that says the registry key has not been created
  13.                         call sub_LogMPScriptEvent (10210, "Error in creation of the Registry Key " & REGKEYPATH & "\" & strRegKeyName & " With Value = " & REGISTRYDEFVALUE , Err, True,strMPScriptName,MOMEVENTLOGERROR)
  14.             End if
  15.             strCurrentDebugModeValue = REGISTRYDEFVALUE
  16. End if The discovery itself : 'Return Data to the Management PackCall oDiscoveryData.AddInstance(oInst)
Call oAPI.Return(oDiscoveryData)
  1.  
  2. Dim intSourceType
  3. Dim strSourceId0
  4. Dim strTargetComputer
  5. Dim strManagedEntityId
  6. intSourceType =
  7. 'Set the arguments for creating the default Management Pack modules.strSourceId = oArgs(0)
  8. strManagedEntityId = oArgs(1)
  9. strTargetComputer = oArgs(2)
  10. 'Create the default management pack classSet oDiscoveryData = oAPI.CreateDiscoveryData(intSourceType, strSourceId, strManagedEntityId)
  11. Set oInst = oDiscoveryData.CreateClassInstance("$MPElement[Name='MyNewMP.CLS1']$")
  12. call oInst.AddProperty("$MPElement[Name='Windows!Microsoft.Windows.Computer']/PrincipalName$", strTargetComputer)
  13. call oInst.AddProperty("$MPElement[Name='System!System.Entity']/DisplayName$", "MyNewMP")
  14. call oInst.AddProperty("$MPElement[Name='MyNewMP.CLS1']/DebugMod$", strCurrentDebugModeValue)
  15.  
  16. call sub_LogMPScriptEvent (10213, "Server "& strTargetComputer & " has been discovered. The debug mode value on this server is set to " & strCurrentDebugModeValue, Err, strCurrentDebugModeValue,strMPScriptName,MOMEVENTLOGINFORMATION)
  17. '-----------------------------------------------------------------------------------------------------
  18. ' PUT YOUR DISCOVERY CODE AFTER
  19. '-----------------------------------------------------------------------------------------------------
  20. 'Return Data to the Management PackCall oDiscoveryData.AddInstance(oInst)
  21. Call oAPI.Return(oDiscoveryData)
'Log the Management Pack default informations with runas account, debugmode and parameters info call sub_LogScriptStartInfo(oArgs, strCurrentDebugModeValue, MOMEVENTLOGINFORMATION, strMPScriptName)



How to use the debugging functionnality will be explained in PART III

End of PART II
Back to Part I                                                                                         Go to Part III


This posting is provided "AS IS" with no warranties.