Wednesday, April 7, 2021

[OpsMgr2019 UR2] SCOM agent update installation failed because access is denied (80070005)

After a long long period with no post, here is a new article with an issue I've encountered on SCOM 2019 UR2. 

All starts with a single administration action in the SCOM console: Approving agent in pending update after having applied patch on Management Servers. I had to apply 'Update for event log channel in System Center Operations Manager 2019 (KB4601269)' to fix CVE-2021-1728 | System Center Operations Manager Elevation of Privilege Vulnerability.

You all know that 'Approving' an Agent in pending required update state results in giving credentials that have administrative right on the targeted agent and click on Update button. This generaly run smoothly. 

In my case, Agent Management task is failing with an access denied message and this generate an error event 10607, source Health Service Modules in the Operations Manager Event Log :

The Operations Manager Server cannot process the install/uninstall request for computer <Computer Name> due to failure of operating system version verification.

Operation: Agent Install

Install account: <Admin Account>

Error Code: 80070005

Error Description: Access is denied. 


I reminded me that we have implemented an Authentication Policy with 'Enforce Policy restriction' mode in Active Directory to restrict usage of NTLM on our Admin Accounts. 

Note: This action has been done because of a prioritized recommendation in the Azure AD Security assessment : Place privileged users in the Protected Users Group. The protected users groups provides additional security, because users can only authenticate using Kerberos (everything else is blocked) and hardenning is applied to the Kerberos authentication used by enforcing AES encryption.

Checking on Domain Controller, I've found that an event 4625 is raised with status "0xc000006e" for the account used in the deployment task at the same date and time the failing task. 
It indicates that the account is trying a network logon (type 3) with authentication Package NTLM, authentication information is valid but some user account restriction has prevented successful authentication. 
The Event description adds that ‘NTLM authentication failed because access control restrictions are required.’ and it gives the name of our Authentication Policy.

NTLM usage block is a known consequence of the Authentication Polices configuration and reverting the Authentication Policy from ‘Enforce Policy Restriction’ to ‘Only audit policy restriction’ for the account use in the deployment task is solving the issue. In audit mode, we always see event in AD for NTLM usage but NTLM authentication stay allowed.

I was thinking that SCOM uses whatever mechanism is available to open a RPC connection at start the installation and then open an SMB connection to copy updates but this shows that without NTLM V2, (NTLMV1 was disabled since year now in our environment), the deployment cannot be successful. 


Are we talking here about a SCOM bug ? 

To be sure I was not missing something in the configuration or having a misconfiguration somewhere, I've open a Microsoft case for this issue. 

I have reproduced the issue in order to create traces and the support confirmed me that 
  1. The deployement start by creating an RPC connection and this connection is well using Kerberos authentication.
  2. Then it uses an SMB session by using IP address and since IP address alway use NTLM, kerberos is not used at this step.
At this step, Microsoft support indicates that no configuration can be done in SCOM as it is by design, however, we can configure the operating System to use Kerberos even for IP adresses by following this article: Configuring Kerberos for IP Adress | Microsoft Docs 

After applying the change, new traces have been done and analysed by Microsoft and we can see in network trace that the SMB is well initiated using Kerberos ! However, we can see later in the trace that we have another RPC session, starts with authentication using Kerberos, but later in the same session, it reverts to NTLM, even though the session was successfully running on Kerberos.


It seems where are here at the limits of the product and the product teams answered that they have never been asked for a similar requirement nor was it tested, thus what makes it unsupported.

To minimize the issue, because agent maintenance or update is not a frequent process (UR3 have to be applied at this time and I'll have the same issue), I've been advised the following:
  1. For NON-Domain Controller servers, use a domain account that have local administrator right. That's true but in my case, the SCOM environment is dedicated to Active directory servers and the few non-DC servers are monitored using System account to avoid managing a domain account that have local admin rights... 
  2. For DC servers, use a Domain admin account that is allowed to used NTLM!!! And if getting an exception is not option, a workarround is to deploy the update through group policy (or similar approach) or event manually. 

I feel that making SCOM working in a full Kerberos environment is very important and other customers also will one day need this (after removing NTLM usage for Security purpose), that's why I've submitted a request on https://systemcenterom.uservoice.com/forums/293064-general-operations-manager-feedback


Feel free to vote for it, the more votes the request gets the more action the product group will take upon it.




This posting is provided "AS IS" with no warranties.

Thursday, October 15, 2015

[OPsMgr 2007 R2][OpsMgr2012] SQL Management pack new version 6.6.2.0 #ManagementPack #OpsMgr #SCOM


New SQL MPs has been delivered by Microsoft yesterday (Oct. the 14th)

MP Name: 'SQL Server 2014'  has been released in version '6.6.2.0' : http://www.microsoft.com/en-us/download/details.aspx?id=42573


Release Date
Changes
October, 2015 (version 6.6.2.0)
·         Fixed performance
·         Added a support for disabled TCP/IP protocol
·         Fixed performance metrics error that may occur on some localized versions of Windows
·         Fixed incorrect performance of Transaction log free space monitor
·         Added new type of events from failed discoveries; added a new reporting rule that collects such events
·         Added overrides to prevent various scripts timeout failure
·         Removed some 1X1 tiles from Summary Dashboards
·         Filestream filegroups are excluded from discovery for now
·         2008/2012 Summary Dashboards tiles were reorganized
·         Added KB for Microsoft.SQLServer.2014.Mirroring.MirroringCommonGroupDiscovery
·         Summary Dashboard: added monitor/performance tiles to class "SQL Server 2014 Mirroring Groups"
·         Some minor fixes


MP Name: 'SQL Server 2005 2008 2012'  has been released in version '6.6.2.0' : https://www.microsoft.com/en-us/download/details.aspx?id=10631

 
Release Date
Changes
October, 2015 (version 6.6.2.0)
·         Added a support for disabled TCP/IP protocol
·         Fixed performance metrics error that may occur on some localized versions of Windows
·         Fixed bugs in monitor tiles on SQL Server Summary Dashboard
·         Fixed incorrect performance of Transaction log free space monitor
·         Added new type of events from failed discoveries; added a new rule that collects such events
·         Added overrides to prevent various scripts timeout failure
·         Removed some 1X1 tiles from Summary Dashboards
·         Filestream filegroups are excluded from discovery for now
·         2008/2012 Summary Dashboards tiles were reorganized


MP Name: 'SQL Server 2014 Replication'  has been released in version '6.6.2.0' : http://www.microsoft.com/en-us/download/details.aspx?id=47720

 
Release Date
Changes
October, 2015 (version 6.6.2.0)
Preset dashboards are updated


MP Name: 'SQL Server 2012 Replication'  has been released in version '6.6.2.0'  : http://www.microsoft.com/en-us/download/details.aspx?id=47721

Release Date
Changes
October, 2015 (version 6.6.2.0)
Preset dashboards are updated

 

 

This posting is provided "AS IS" with no warranties.

Thursday, October 8, 2015

[OpsMgr 2012 R2] Changing Target Group in Process Monitoring is failing

Today, I've encountered an issue with a process monitoring.
The case has been detected because the targeted group that was enabling my process monitor has been deleted in SCOM... and the result is an error message when editing the Process Monitor


Clicking OK is opening the properties and we can see that targeted group field is empty...


 You are also allowed to clik on the [...] button to select a new group.

 
The bug is that any activities on the targeted group field is not seen as a change on the monitor configuration and the Apply button is always greyed !At this step, you cannot save the change and also your new targeted group cannot be saved.

A work-arround is to edit a field in an other pane, for example in the Running Processes pane and you will see that the Apply button has been activated.


 
 Applying the change at this step will save your new targeted group change.



You can easyly reproduce this with no group deletion - just try to change the targeted group by an other and you will see that the Apply Button is always in grey state meaning the the change is not 'seen' by SCOM and you will not be able to save the new conf.


By the way, Microsoft has been informed and a bug will be openned to the developper team... to be followed !



This posting is provided "AS IS" with no warranties.

Wednesday, July 1, 2015

[OpsMgr 2007 R2] [OpsMgr 2012] Microsoft System Center Management Pack for SQL Server #SQL #OpsMgr


We just have in place the last version of the SQL Monitoring in our productions environment and Microsoft has released a new one. New version is now 6.6.0.0 and please find here the different link on the Microsoft site.  

 
 



- 'SQL Server 2005 2008 2012'  Version: '6.6.0.0'  Released on: '12-15-2014'
- 'SQL Server 2014'  Version: '6.6.0.0'  Released on: '12-15-2014'
- 'SQL Server 2008 Reporting Services (Native Mode)'  Version: '6.6.0.0'  Released on: '11-4-2014'
- 'SQL Server 2012 Reporting Services (Native Mode)'  Version: '6.6.0.0'  Released on: '11-4-2014'
- 'SQL Server 2012 Analysis Services'  Version: '6.6.0.0'  Released on: '11-4-2014'
- 'SQL Server 2008 Analysis Services'  Version: '6.6.0.0'  Released on: '11-4-2014'
- 'SQL Server 2014 Reporting Services (Native Mode)'  Version: '6.6.0.0'  Released on: '11-4-2014'
- 'SQL Server 2014 Analysis Services'  Version: '6.6.0.0'  Released on: '11-4-2014'

 
The Management pack for SQL Server provides the capabilities for Operations Manager 2007 R2 and Operations Manager 2012 to discover SQL Server 2005, 2008, 2008 R2, and SQL Server 2012. It monitors SQL Server components such as database engine instances, databases, and SQL Server agents.

The monitoring provided by this management pack includes performance, availability, and configuration monitoring, performance data collection, and default thresholds. You can integrate the monitoring of SQL Server components into your service-oriented monitoring scenarios.

In addition to health monitoring capabilities, this management pack includes dashboard views, diagram views and extensive knowledge with embedded inline tasks, and views that enable near real-time diagnosis and resolution of detected issues.

Important Prerequisite Notes:
Clusters: In order to ensure that all monitoring works correctly for clustered instances of SQL Server ensure that your OpsMgr agents on the physical nodes of the cluster are running either OpsMgr 2007 R2 or OpsMgr 2007 SP1 with the most recent cumulative update for OpsMgr 2007 SP1 applied or OpsMgr 2012. 

Changes in June 2015 Update of the SQL 2005,2008,2012 and 2014 Management pack

SQL Server Management Pack version 6.6.0.0 includes the following changes:
·         Dashboards were replaced with the new ones
·         Components of replication functionality are deprecated and disabled by default
·        SPN monitor now correctly handles disjoined namespaces
·         Added support for filegroups containing filestreams and partition schemes
·         Memory Consumption monitor has been fixed
·         Upgradeability from 6.4.1.0 version is supported
·         Added CPU Usage monitor and rule for SQL Server 2005
·         Added ConsecutiveSamples Condition to the Buffer Cache Hit Ratio and Page Life Expectancy monitors
·         AlwaysOn discovery was reworked
·         Minor fixes.


For more information, read the détails on the different Microsoft download pages.
 

This posting is provided "AS IS" with no warranties.

Monday, June 8, 2015

[OpsMgr 2007 R2][OpsMgr 2012] Skype for Business Server 2015, Management Pack #SkypeForBusiness #Opsmgr2012 #OpsMgr2007R2

 
Skype for Business Server 2015, Management Pack contain comprehensive monitoring tools including:
  • End-to-End scenario availability from various locations.
  • Call reliability and quality perceived by real users.
  • Component health and performance.
Active Monitoring Management pack
  • Geographically distributable end-to-end scenario validation for Skype for Business Server 2015 deployments. This includes coverage for internal, remote and branch office users.
  • Increased scenario coverage in areas like AV Edge Connectivity and Exchange Unified Message Connectivity.
  • Enhanced troubleshooting logs to streamline root cause analysis of failures.
Component and User Management Pack
  • Increased reliability monitoring of real end user calls and conferences.
  • Integrated media quality alerts driven from Quality of Experience (QoE) data reported by client applications.
  • Full event and performance monitoring for all Skype for Business Server 2015 roles.


 
 
 
System Requirements
  • Supported Operating System
  • Windows Server 2008 R2, Windows Server 2012 R2
System Center Operations Manager:
  • System Center Operations Manager 2007 R2
  • System Center Operations Manager 2012; 64-bit agents only 

This posting is provided "AS IS" with no warranties.

Friday, May 22, 2015

[Free Ebook] Free ebook: Microsoft System Center Operations Manager Field Experience #Free #Ebook

Microsoft System Center Operations Manager Field Experience (ISBN 9780735695825), by Danny Hermans, Uwe Stürtz, Mihai Sarbulescu; Mitch Tulloch, Series Editor is downloadable for free.
 
Download all formats (PDF, Mobi and ePub) at the Microsoft Virtual Academy.
 
 

Introduction

If you’re responsible for designing, configuring, implementing, or managing a Microsoft System Center Operations Manager environment, then this book is for you. This book will help you understand what you can do to enhance your Operations Manager environment, and will give you the opportunity to better understand the inner workings of the product, even if you are a seasoned Operations Manager administrator.
This book assumes that you have a deep working knowledge of the Operations Manager product and its concepts, that you understand the concept of management packs, and that you are basically familiar with Microsoft Azure as an infrastructure-as-a-service platform. This is a book about best practices, design concepts, how-tos, and in-depth technical troubleshooting. It covers the role of the Operations Manager product, the best practices for working with management packs, how to use the reporting feature to simplify managing the product, how to thoroughly troubleshoot, and how to use and install Operations Manager in a Microsoft Azure Public Cloud environment.
 

About the companion content

The companion content for this book can be downloaded from the following page:
http://aka.ms/OpsMgrFE/files
The companion content includes the following:
  • The SQL query in Chapter 1 that you can run in SQL Server Management Studio to determine which collation settings you are using
  • The series of commands used in the example in Chapter 2 to run workflow tracing manually
  • The Windows PowerShell script used in Chapter 4 to view all TLMEs that exist order per resource pool and per current owning pool member (management server)
  • The various SELECT queries included in Chapter 4
  • A PDF file titled HealthService Event Reference that provides information about the events that Operations Manager can log to its event log from the HealthService features.

Acknowledgments

We would like to thank Daniele Muscetta, Microsoft Program Manager for Azure Operational Insights, for his review and comments on the Azure Operational Insights section of Chapter 5; Stefan Stranger, Microsoft Senior Premier Field Engineer, for the review of and his input on the remainder of Chapter 5; and Danny’s loving wife, Vita Martinsone, for the pre-editing and formatting of our work.

This posting is provided "AS IS" with no warranties.

[Orcherstrator] System Center Orchestrator Migration Toolkit Version 7.3 #Azure #Orchestrator #Toolkit

A collection of tools for migrating integration packs, standard activities, and runbooks from System Center 2012 – Orchestrator to Azure Automation and Service Management Automation. 


The System Center Orchestrator Migration Toolkit consists of the following tools.

Integration Pack Converter
This tool converts integration packs that were created using the Orchestrator Integration Toolkit to integration modules based on Windows PowerShell that can be imported into Azure Automation or Service Management Automation. Using the tool’s wizard, you can select the activities in the integration pack that will be converted to cmdlets in the integration module. Placeholders are created for monitor activities that are not supported in Azure Automation or Service Management Automation.

Standard Activities module
Integration module that contains all of the standard activities used in Orchestrator Runbooks that can be imported into Azure Automation or Service Management Automation. This module must be installed in your environment prior to importing any runbook converted with the Runbook Converter.

Runbook Converter (coming soon)
This tool converts Orchestrator runbooks into graphical runbooks that can be imported into Azure Automation.



System Requirements
Supported Operating System
  • Windows Server 2012 R2, Windows Server 2012 R2 Datacenter, Windows Server 2012 R2 Standard
  • Supported Operating System
  • Windows Server 2008 R2, Windows Server 2012, Windows Server 2012 R2
Recommended Configuration for System Center Orchestrator Migration Toolkit
  • RAM – minimum 1GB recommended 2GB
  • Disk space minimum 200MB
  • CPU Dual Core or better
  • Windows Server 2012 R2
  • .Net 4.5

This posting is provided "AS IS" with no warranties.