How to configure SFM, WBEM, SMH, RSP, and SIM all together?
The whole idea of the RSP (Remote Support Pack) implementation is to incorporate service tools that notify the customer when a significant system event has occurred or may occur. You can configure and customize alert notification status for servers, storage, network and selected multivendor devices.
This article covers all the necessary steps to configure an HPUX server, so in a case of a failure it reports an fault or event in a timely manner to the SIM server and to HP's backend using the WBEM protocol.
Contents
- 1 Checking Requirements for SysFaultMgmt
- 2 Installation SysFaultMgmt Software
- 3 WBEM & SFM check and setup
- 4 Setup System(s) for Remote Support Pack
- 5 Configuration Steps (RSP Related not related to HPUX)
- 6 Check contract status (RSP Related not related to HPUX)
- 7 Creating WBEM subscription
- 8 Verifying WEBES Subscription (RSP Related not related to HPUX)
- 9 Verifying status on HPUX host
- 10 Verifying status in SIM
- 11 Installation of the Advanced Configuration Collector (ACC) (RSP Related not related to HPUX)
- 12 SIM Server Certificate
- 13 SIM Client Certificate (only used if a non-privileaged user will not be used)
- 14 Preparation for the collection (RSP Related not related to HPUX)
- 15 Schedule and start a collection (RSP Related not related to HPUX)
- 16 Useful Information
- 17 RSP related processes
- 18 HPUX directories and logs
- 19 Windows directories and logs
- 20 Troubleshooting
Checking Requirements for SysFaultMgmt
From the HPUX perspective, System Fault Management (SFM) is a collection of tools used to monitor the health of HP servers and receive information about hardware such as memory, CPU, power supplies, and cooling devices. SFM operates in the Web-Based Enterprise Management (WBEM) environment. This is the last software to be installed in the system after meeting all of its requirements.
SFM exists for 11.11, 11.23 and 11.31 HPUX OS versions.
Here is a set of useful links:
- SFM Tables of versions
http://docs.hp.com/en/diag/sfm/sfm_table.htm
For each one of the versions released, there is a Release Notes, in which you can check the Requirements for each one of them. Following is the Release Notes for the latest SFM version for each flavor of the HPUX OS:
- System Fault Management A.04.01.01.02 Release Notes, HP-UX 11i v1 April 2009
http://www.docs.hp.com/en/diag/sfm/sfm_0904_1111.htm (Check the System Requirements Section)
- System Fault Management B.07.01.01.03 Release Notes, HP-UX 11i v2 May 2009
http://www.docs.hp.com/en/diag/sfm/sfm_0905_1123.htm (Check the System Requirements Section)
- System Fault Management C.06.00.07.01 Release Notes, HP-UX 11i v3 September 2009
http://www.docs.hp.com/en/diag/sfm/sfm_0909_1131.htm (Check the System Requirements Section)
Installation SysFaultMgmt Software
The SysFaultMgmt Software comes with the installation CD, and also can be downloaded from www.software.hp.com.
The key products that you need to make sure that are installed before installing SFM are:
- EMS https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=EventMonitoring
- EMS B5736DA (EMS High Availability Monitors, recommended)
- OnlineDiag (STM) https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=B6191AAE (11.11 and 11.23) https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=DiagProdCollection (bundled in this depot in 11.31)
- Openssl https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=OPENSSL11I
- WBEMServices https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=OPENSSL11I
- SMH https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=SysMgmtWeb (recommended but not required)
- Additional requirements specified in SFM Release Notes.
1) For the hp.com depot do:
#swinstall -s <depot_path> SysFaultMgmt
2) For the OE Media:
# swinstall -x autoselect_dependencies=true -x enforce_dependencies=true -s /tmp/cdrom SysFaultMgmt
WBEM & SFM check and setup
After installing all products, these extra checks and steps are necessary.
1) Check that the correct version of the cimserver is running:
# /opt/wbem/sbin/cimserver –v A.02.07
2) Check that the SFMProvider Module is running fine:
# /opt/wbem/bin/cimprovider -l –s MODULE STATUS …… SFMProviderModule OK ……
3) Check that the providers are properly registered with the SFM Provider Module
# /opt/wbem/bin/cimprovider -l -m SFMProviderModule
CPUProvider
CPUStatusProvider
EMSWrapperProvider
SFMIndicationProvider
EventIndicationConsumer
MemoryProvider
MemoryStatusProvider
DiskProvider
DiskStatusProvider
StateChangeIndicationProvider
ChassisProvider
CoolingStatusProvider
PowerStatusProvider
ThermalProvider
VoltageProvider
MPProvider
MPStatusProvider
FirmwareRevisionProvider
HPUX_ControlProvider
FMDProvider
HealthStateProvider
EMArchiveConsumer
EMEmailConsumer
emdprovider
SubscriptionConfigAssociationProvider
ThrottlingConfigInstanceProvider
WBEMToEMSConsumer
EnclosureProvider
4) Check that EMS is enabled:
# /etc/opt/resmon/lbin/monconfig
5) Check that SFM is used for hardware monitoring (this is only good for 11.23 and 11.31 HPUX versions):
# /opt/sfm/bin/sfmconfig -w –q
EMS hardware monitors are disabled & SysFaultMgmt is monitoring devices.
If EMS is still enabled for monitoring, switch to SFM using the command:
# /opt/sfm/bin/sfmconfig –w -s
6) Check that the HP-defined filters are setup:
# /opt/sfm/bin/sfmconfig -m list
Filter Name : General Filter
Filter Type : HP Defined Filter
Filter Unique Identifier : 1
Filter Query : Select * from HP_DeviceIndication
Filter Query Language : WQL
Filter Source Namespace : root/cimv2
Filter Description : General Device Indications.
Filter State : Enabled Filter State <<<< Is setup!!!
Filter Last Operation : No Operation
--------------------------------------------------------
If the HP-defined filters are not listed:
#/opt/sfm/bin/sfmconfig –m enable –n “General Filter” –u 1 –t HP
7) Change the startup mode for the System Management Homepage so that the autostart url mode is set to OFF and start on boot mode is set to ON:
#/opt/hpsmh/lbin/hpsmh stop #/opt/hpsmh/bin/smhstartconfig -a off -b on #/opt/hpsmh/lbin/hpsmh start #/opt/hpsmh/bin/smhstartconfig HPSMH 'autostart url' mode.........: OFF HPSMH 'start on boot' mode.........: ON Start Tomcat when HPSMH starts.....: OFF
8) Make sure that EMS framework and SFM is able to send test events to /var/opt/resmon/log/event.log and evweb (SFM event handler):
- For systems that don't support IPMI and are running 11.11 use this command to send test events:
# /etc/opt/resmon/lbin/send_test_event -a -v disk_em (you can use dm_memory, dm_core, and other monitors also)
- For systems that support IPMI, or are running 11.23 and 11.31 you can use the previous command, or use the following:
# /opt/sfm/bin/sfmconfig -t -a (make sure that sfmconfig -w -q reports "SysFaultMgmt is monitoring devices")
After sending the test events, check /var/opt/resmon/log/event.log to see if the events reached the file and also check evweb:
# /opt/sfm/bin/evweb eventviewer -L
NOTE: If the events reached evweb, there is a big chance that the events will make it to the SIM server later on. By reaching evweb we are testing that EMS generates the events and also the EMS Wrapper is translating the events into WBEM events, which are analyzed by SFM and displayed in evweb.
Once you have made sure that the events reach evweb, you can proceed to add the server to SIM server and discover it, identify it and subscribe it to WBEM events.
Setup System(s) for Remote Support Pack
After installation of SysFaultMgmt Web the systems needs to be discovered by SIM. To do this, start the SIM:
A) https://<Sim_Server>:50000 and navigate to Options Discover. If you want to discover single IP addresses, select “Manual”.
B) Enter your IP Address and press “Add System”.
If you want to discover a range of IP addresses choose “Automatic” and press “New”
C) Enter a name for you discovery, uncheck the “Automatically execute discovery every:” check box, enter you IP address range and press “OK”. Select your discovery task and press “Run Now”.
The discovery will take a while, please wait until the task is finished.
D) When the task is finished, navigate to “All Systems” and check that the system(s) are listed and properly discovered.
If the system is not discovered properly, navigate to: ->Options ->Identify System ->“Run Now”
Wait for the status to be “Completed” and check if the system is now properly discovered.
It's recommended, that for every configuration change that is made in the HPUX side, an Identify is re-run in the SIM server for that specific server.
NOTE: This part is related directly with RSP. Remember that there is a team specialized in this.
Set System properties (single servers)
The next configuration step is to set the system properties for the system(s).
A) To do this for a single system, click on the system name in the “All Systems” section
+ "Tools & Links" +-+ "Edit System Properties"
- Enter | Verify Company and Contact Information
- Add serial and part number if not already listed
B) Create System properties task (for multiple systems)
C) To enter the same company and contact information for several systems and every new system discovered, select: ->Options ->System Properties ->Set System Properties
D) From the pull-down menu select the system type that will have the same contact – if the contact for all systems is the same select “All Systems”. Press “Apply” and “Next>”.
Verify the selection, change it if needed and press “Next>”.
On the next screen, scroll down to the “Contract and Warranty Information” section.
For the “System Site Information, either select a site from the pull-down menu or press “Manage Sites…” To add a new site press “New…” and fill out the required fields.
Proceed with the “Manage Contacts” section in the same way:
When all date has been entered, use the “Schedule” Button to continue.
To verify the contract status of the system, navigate to “All Systems” and select your device: ->Options ->Contract and Warranty Data Collection ->Press “Run Now” Wait for the task to be completed.
If the entitlement worked fine, check the status in the Remote Support Pack.
To do this, navigate to
->Options
->Remote Support Configurations and Server
Select the “Entitlement” tab
If the system is not listed, press the “Synchronize RSE” button. If the entitlement status is not correct, select the system. From the “Action List” select “Entitle checked” and press “Run Action”.
Creating WBEM subscription
The WBEM subscriptions are needed, in order for all the events that are generated in the HPUX node, to be sent to the systems (like a SIM server)that are subscribed to WBEM events with this server.
In order to see to what systems the HPUX node is subscribed to, you need to run the following command:
# /opt/sfm/bin/evweb subscribe -L -b external (to see all external subscriptions)
Filter Name Handler Name Query Destinat Destination Url
====================== ======================================= ======================================== ======== ==========================================
HP_General Filter@1_V1 localhost/CIMListener/EMArchiveConsumer Select * from HP_DeviceIndication UNKNOWN
HPSIM_noaadc-hpsim_0 HPSIM_noaadc-hpsim select * from HP_DeviceIndication CIMXML https://147.154.10.132:50004/cimom/listen1
HPSIM_noaadc-hpsim_1 HPSIM_noaadc-hpsim select * from HP_ThresholdIndication CIMXML https://147.154.10.132:50004/cimom/listen1
Usually most of the information in this output is just a reference, and you should not need to try to decipher all fields. You just have to make sure that the output is showing the name of the SIM server and it's ip connecting through address using port 50004. The port 50004 is the one used to send WBEM events to the SIM server. So in summary, in the previous example what this indicates is, that whenever an event occurs, it will be detected by EMS, SFM and also sent to the SIM server via port 50004 to address 147.154.10.132.
This subscriptions are created when you subscribe the HPUX node in the SIM server to WBEM events.
Usually the WBEM protocol, traveling from SIM server to the HPUX node, needs to authenticate in order to establish a communication using port 5989 (WBEM port). One needs to provide access to the protocol so it can gather the necessary information from HPUX node and provide it to the SIM server. Usually a non-privileaged user is used for that. Most the access will be given to read all the information from /var/opt/wbem/repository directory.
Creating WBEM Subscription using a non-privileged user
If another account than the root account should be used for WBEM, setup an account for the WBEM user using “SAM”.
A) Navigate to:
->Accounts for Users and Groups ->Users
-- From the menu select “Actions” => Add -- Enter the needed data and press ok.
Or use the CLI command:
- useradd –m <user name>
- passwd <user name>
B) Now add read and write authorizations for the new WBEM user to each of the namespaces under /var/opt/wbem/repository:
-- root/cimv2 -- root/PG_InterOp -- root/PG_Internal: -- root/cimv2/PG_npar: -- root/cimv2/PG_vpar: -- root/cimv2/PG_hpvm:
Using the command cimauth:
#cimauth -a -u <username> -n root/cimv2 -R -W #cimauth -a -u <username> -n root/PG_InterOp -R -W #cimauth -a -u <username> -n root/PG_Internal -R –W #cimauth -a -u <username> -n root/cimv2/npar -R -W >>> This is needed (a must) if Npars are being used. #cimauth -a -u <username> -n root/cimv2/vpar -R -W >>> This is needed (a must) if Vpars are being used. #cimauth -a -u <username> -n root/cimv2/hpvm -R -W >>> This is needed (a must) if hpvm are being used.
NOTE: The vpar and npar namespaces are added by the vpar and npar provider respectively. And the hpvm namespace is added by the HPVM Software when installed (HPVM Guest specially).
To check that all authorizations are set properly execute:
#cimauth -l <username>, root/cimv2, "rw" <username>, root/PG_InterOp, "rw" <username>, root/PG_Internal, "rw" <username>, root/cimv2/npar, "rw" <username>, root/cimv2/vpar, "rw" <username>, root/cimv2/hpvm, "rw"
To be able to do a WBEM event subscription with a non privileged account this has to be enabled as well.
C) Checked the current settings using the command:
#cimconfig -l -c
enableAuditLog=false
sslClientVerificationMode=optional
enableSubscriptionsForNonprivilegedUsers=false
shutdownTimeout=30
authorizedUserGroups=
enableRemotePrivilegedUserAccess=true
enableHttpsConnection=true
enableNamespaceAuthorization=false
enableHttpConnection=false
D) If the enableSubscriptionsForNonprivilegedUsers is set to false, this needs to be changed to true:
#cimconfig -s enableSubscriptionsForNonprivilegedUsers=true -p #cimconfig -s enableNamespaceAuthorization=false -p #cimserver –s #cimserver
E) After this has been done, the credentials for the system have to be set first. Therefore select the system (if more than one system has the same credentials, select all of them) in the “All Systems” section and navigate to: ->Options ->Protocol Setting ->System Protocol Setting
For the WBEM settings select “Update values for this protocol” and “Use values specified below”, then enter the user and password.
Select “Run Now” and wait for the task to be completed.
F) When this has been successfully completed, you need to subscribe to WBEM events. To do again select the system(s) in the “All systems” section and choose ->Options ->Events ->Subscribe to WBEM Events
Select “Next” and on the second page confirm the task with “Run Now”. Wait for the task to be completed – this may take a while especially for PA RISC systems.
G) To verify the WBEM subscription from the HPUX device use the following command:
#/opt/sfm/bin/evweb subscribe -L -b external
Filter Name Handler Name Query Destinat Destination Url
HPSIM_simserver_0 HPSIM_simserver select * from HP_DeviceIndication UNKNOWN
HPSIM_ simserver_1 HPSIM_ simserver select * from HP_ThresholdIndication UNKNOWN
HPSIM_TYPE_1_ simserver_0
HPSIM_TYPE_1_ simserver_client SELECT * FROM CIM_AlertIndication CIMXML https://simserver:50004/cimom/listen1
HPWEBES_simserver_Filter_HP_AlertIndication HPWEBES_simserver_Handler_HP_AlertIndication select * from HP_AlertIndication UNKNOWNVerifying status in WEBES
A) To verify the WBEM subscription in WEBES start the System Event Analyzer either using: ->Start ->Programs ->Hewlett-Packard Service Tools ->System Event Analyzer ->System Event Analyzer
Or
Starting a web browser using the URL: https://simserver-ip:7906
Login with <any name>-adv to get privileged rights. Select each system and confirm the data.
NOTE: Do not change any data within WEBES. If anything is missing/wrong go back to SIM and enter / change the data in SIM.
Verifying status on HPUX host
There are two types of providers , the EMS wrapper providers and native providers. For some hardware we still use EMS to detect the event but this is forwarded to an EMS wrapper provider which is part of SFM. And for some hardware components there are already native providers.
EMS wrapper provider:
A) Check which EMS Monitors are running using the command:
#ps -ef|grep -i stm
B) Send a test event using oner of these monitor
#/etc/opt/resmon/lbin/send_test_event –v <monitor name> f.e. #/etc/opt/resmon/lbin/send_test_event –v disk_em
Note: Check that a proper Resource Class/Hardware is listed for this monitor!
C) Read through the evweb output if the test event is listed – the event # of a test event is always 103:
#/opt/sfm/bin/evweb eventviewer -L EvArchNo Severity Event # Event Category Archive Time Summary ========= =========== ============== ============== ============== ============== 10 Information 103 Unknown 2007-07-07 12: This is a t...
D) To get more info about that event use the following command:
#/opt/sfm/bin/evweb eventviewer –E –n <EvArchNo>
#/opt/sfm/bin/evweb eventviewer –E –n 10
Event Time...........: Mon NOV 09 12:33:30 2008
Severity.............: 2
Provider.............: DiskIndicationProvider
Event #..............: 103
System...............: server.deu.hp.com
Summary:
Disk at hardware path 0/2/1/0.0.0.0.0 : Test event
Description of Error:
This is a test message from the monitor to test the communication path from the monitor
to the notification mechanism.
Probable Cause / Recommended Action:
No action required.
SFM native provider (not available for 11.11)
# /opt/sfm/bin/sfmconfig -t -a Sending test event for fpl_em monitor. Sending test event for ia64_corehw monitor. Sending test event for memory monitor. Sending test event for processor monitor. Sending test event for backbone monitor.
Note: The number and type of monitors could vary depending on the OS and the System type.
Verifying status in SIM
A) To verify the receipt of test events in HP SIM navigate to “All Events”:
B) To check if they are as well transmitted to the RSP, navigate to:
->Events ->Shared ->Service Events ->All HP Service Events
HP Service Events (processed by WEBES and submitted to HP):
The Advanced Configuration Collector is supported for proactive customers only – ACC can be enabled only for systems listed with a proactive contact in HP SIM. The Software is available through the Software Update Manager.
A) To check whether the latest version of this bundle has been downloaded start the Remote Support Software Manager on you CMS (by default there is an icon on the desktop).
B) Select “Packages” and navigate to “Service Essentials - Remote Support Pack end-point server software for Proactive Services” – there is a package called “Advanced Configuration Collector installation depot for HP-UX 11.11-11.31 servers A.05.xx.xxx.xxx.
Make sure it is installed, which means downloaded to the CMS.
C) Change to the directory
C:\Hewlett-Packard\BusinessSupport\UC\ACC\HPUX\
and copy the depot rs-acc-hpux-11.X-ALL-05.xx.xxx.xxx.depot.gz to your HPUX system (if you used the internal depot this depot should already be installed).
Installing Software
A) To install the Advanced Configuration Collector software, first make sure that the HP Webservices components are installed on the HPUX server:
# swlist -l product|grep hpuxws
hpuxwsAPACHE B.2.0.59.01 HP-UX Apache-based Web Server
hpuxwsTOMCAT B.5.5.23.00 HP-UX Tomcat-based Servlet Engine
hpuxwsWEBMIN A.1.070.10 HP-UX Webmin-based Admin
hpuxwsXML A.2.03 HP-UX XML Web Server Tools
B) If it is not installed, install it from the OE Media or use the internal swdepot.
C) To install the Configuration Collector, uncompress the file first and use swinstall to install it:
#gunzip rs-acc-hpux-11.X-ALL-05.xx.xxx.xxx.depot.gz #swinstall –s <depot path>/rs-acc-hpux-11.X-ALL-05.xx.xxx.xxx.depot \*
D) Verify that the software is installed:
# swlist -l product | grep -i uc-acc UC-ACC B.05.05.021.116 HP Remote Support Advanced Configuration Collector
SIM Server Certificate
To import the SIM Server certificate on the client system,
A) Open the System Management Homepage https://<client ip>:2381 and login using root user credentials.
B) Navigate to
->Settings ->Security ->Trusted Management Servers
C) Enter the CMS IP Address in the blank field on the bottom and press
“Add Certificate From Server”. On the next page press “Add Certificate to Trust List”.
SIM Client Certificate (only used if a non-privileaged user will not be used)
This step can be skipped if root credentials are used with the “configure & repair agent” task.
A) To import the client certificate to the SIM, the file /etc/opt/hp/sslshare/cert.pem needs to be imported in case another user than root has been selected for the WBEM subscription (selecting “Configure and Repair Agents” for subscription).
B) To do this copy this file to your CMS and in SIM navigate to:
->Options ->Security ->Certificates ->Trusted Certificate
C) Choose “Always Accept” and press “Import”. Enter the location for the cert.pem file and press “OK”.
A) To make sure that the ACC software is seen, start again an “Identify System”.
B) Navigate to “All System” and select the systems, from the menu choose
->Options ->Identify System And press “Run Now” - wait for the task to complete successfully.
C) After the successful identification, verify that the server is seen under:
->Options ->Remote Support Configuration and Services ->Server Services
Note: It might take several minutes before the server shows up. Keep in mind that this feature is available only for servers with proactive contract!
A) To schedule and start a server collection, again first select you system(s) under “All Systems” and then navigate to:
->Options ->Status Polling ->HP Service Server Configuration Collection
B) Select „Schedule“ to schedule to collection e.g. once a week at a time most convenient and select „Run Now”.
C) Wait for the task to finish – this might take up to 20 minutes. You should see a success message within HP SIM. In addition you can check the logfile c:\Program Files\Hewlett-Packard\UnifiedCollector\log\uc.log
Useful Information
Diagnostic and SysFaultMgmt related processes
Diagnostic Processes:
# ps -ef|grep -i diag root 1772 1 0 Feb 28 ? 4:12 /usr/sbin/stm/uut/bin/sys/diagmond root 2305 1772 0 Feb 28 ? 1:07 diaglogd
EMS Monitor (some should be listed – which one depends on the hardware), the WbemWrapperMonitor has to be there:
# ps -eaf|grep -i monitor
root 2415 1 0 Jul 15 ? 0:02 /usr/sbin/stm/uut/bin/tools/monitor/disk_em
root 2187 1 0 Jul 15 ? 0:00 /usr/sbin/stm/uut/bin/tools/monitor/WbemWrapperMonitor
root 2854 1 0 Jul 15 ? 0:00 /usr/sbin/stm/uut/bin/tools/monitor/sysstat_em
Event Archive Database service:
# ps -eaf | grep sfmdb
sfmdb 23605 23604 0 22:56:43 pts/0 0:00 postgres: stats collector process
sfmdb 23601 1 0 22:56:43 pts/0 0:00 /opt/sfmdb/pgsql/bin/postmaster -i -D /var/opt/sfmdb/pgsql
sfmdb 23604 23601 0 22:56:43 pts/0 0:00 postgres: stats buffer process
Cimserver Processes:
# ps -eaf | grep cimserver
root 1673 1 0 Jul 15 ? 4:15 /opt/wbem/lbin/cimserver
root 1674 1673 0 Jul 15 ? 0:01 /opt/wbem/lbin/cimservera
root 2349 1 0 Jul 15 ? 0:00 /opt/wbem/lbin/cimserverd
# cimprovider -ls
MODULE STATUS
OperatingSystemModule OK
ComputerSystemModule OK
ProcessModule OK
IPProviderModule OK
DNSProviderModule OK
NTPProviderModule OK
NISProviderModule OK
SDProviderModule OK
AmgrAgentProviderModule OK
SGProvidersModule OK
EMSHAProviderModule OK
HPUX_ProviderModule OK
HPUXLVMProviderModule OK
HP_NParProviderModule OK
HP_ResParProviderModule OK
HPUXRAIDSAProviderModule OK
HPUXSASProviderModule OK
HPUXSCSICSProviderModule OK
HPUXSCSIProviderModule OK
HP_VParProviderModule OK
HPUXFCCSProviderModule OK
HPUXFCIndicationProviderModule OK
HPUXFCProviderModule OK
FSProviderModule OK
HPUXIOTreeIndicationProviderModule OK
HPUXLANProviderModule OK
HPUXLANCSProviderModule OK
HPUXLANIndicationProviderModule OK
HP_iCODProviderModule OK
HP_iCAPProviderModule OK
HP_GiCAPProviderModule OK
HP_UtilizationProviderModule OK
HPVMProviderModule OK
IOTreeModule OK
SFMProviderModule OK
# cimprovider -lm SFMProviderModule
CPUProvider
CPUStatusProvider
EMSWrapperProvider
SFMIndicationProvider
EventIndicationConsumer
MemoryProvider
MemoryStatusProvider
DiskProvider
DiskStatusProvider
StateChangeIndicationProvider
ChassisProvider
CoolingStatusProvider
PowerStatusProvider
ThermalProvider
VoltageProvider
MPProvider
MPStatusProvider
FirmwareRevisionProvider
HPUX_ControlProvider
FMDProvider
HealthStateProvider
EMArchiveConsumer
EMEmailConsumer
emdprovider
SubscriptionConfigAssociationProvider
ThrottlingConfigInstanceProvider
WBEMToEMSConsumer
EnclosureProvider
The Remote Support Pack Client has only one service running on the CMS. The service appears on the Windows OS as “HP ISEE Client”, though it is functionally the RSP client and has no relationship with the ISEE product.
net stop “HP ISEE Client”
net start “HP ISEE Client”
or
net stop iseeclientsvc
net start iseeclientsvc
HPUX directories and logs
The install directory for System Fault Manager is /opt/sfm
The log file for the System Fault Manager is
/var/opt/sfm/log/sfm.log 11iv2 and 11iv3
/opt/opt/sfm/log/sfm.log 11iv1
WBEM is installed in the directory /opt/wbem
The logfiles are located at /var/opt/wbem/log
Windows directories and logs
The install directory for the Remote Support Pack is: C:\Program Files\Hewlett-Packard\RemoteSupport
The log files are located at: C:\Program Files\Hewlett-Packard\RemoteSupport\logs
The Unified Collector is installed at: C:\Program Files\Hewlett-Packard\UnifiedCollector
The log files are located at: C:\Program Files\Hewlett-Packard\UnifiedCollector\log
The SIM itself is installed at: C:\Program Files\HP\Systems Insight Manager
The log files are located at: C:\Program Files\HP\Systems Insight Manager\logs
And the Software Update Manager at:
- C:\Program Files\HP\SWMAgent HP SIM 5.2
- C:\Program Files\ Hewlett-Packard \SWMAgent HP SIM 5.1
The Remote Support Client writes to the following logs by default:
- c:\Program Files\Hewlett-Packard\RemoteSupport\logs\iseeclient.log
The iseeclient.log file is the first place to look if there is a problem with the client service or a registration incident.
The log file
- c:\Program Files\Hewlett-Packard\RemoteSupport\logs\submitIncident.log
records test events and other communications that are prompted manually by the user or automatically through scripts run at the HP Support Center.
The log file
- c:\Program Files\Hewlett-Packard\RemoteSupport\logs\submitProxiedIncident.log
records hardware events and other communications that result from OSEM event submission to the Remote Support Client, as well as OS registration and all entitlement requests sent to the HP Support Center.
The log file
- c:\Program Files\Hewlett-Packard\RemoteSupport\logs\submitProxiedData.log
records Remote Support Configuration Collections (RSCC) related information.
Troubleshooting
On the Managed System.
A) Check that the cimserver is running:
# ps -eaf | grep cimserver
root 1673 1 0 Jul 15 ? 4:15 /opt/wbem/lbin/cimserver
root 1674 1673 0 Jul 15 ? 0:01 /opt/wbem/lbin/cimservera
root 2349 1 0 Jul 15 ? 0:00 /opt/wbem/lbin/cimserverd
B) If it is not start it, do a
#cimserver
To restart the cimserver
#cimserver –s to stop the service #cimserver to start the service
C) Check the version of the cimserver:
# /opt/wbem/sbin/cimserver –v A.02.05.08
If it is not the correct version, install the correct one using swinstall.
D) Check the health of the WBEM services:
# osinfo OperatingSystem Information Host: server.deu.hp.com Name: HP-UX Version: B.11.23 UserLicense: Unlimited user license Number of Users: 1 users Number of Processes: 157 processes OSCapability: 64 bit LastBootTime: Jul 15, 2007 16:27:35 (00200) LocalDateTime: Jul 17, 2007 23:41:27 (00200) SystemUpTime: 198832 seconds = 2 days, 7 hrs, 13 mins, 52 secs
E) Check that the SFMProviderModule and the EMSHAProviderModule is in status “OK” and are present (this 2 are a must to subscribe a managed node to wbem events) and its providers are registered:
# cimprovider -ls
MODULE STATUS
OperatingSystemModule OK
ComputerSystemModule OK
ProcessModule OK
IPProviderModule OK
DNSProviderModule OK
NTPProviderModule OK
NISProviderModule OK
SDProviderModule OK
AmgrAgentProviderModule OK
SGProvidersModule OK
EMSHAProviderModule OK <--------------------- Must be present
HPUXLVMProviderModule OK
HP_NParProviderModule OK
HP_ResParProviderModule OK
HPUXRAIDSAProviderModule OK
HPUXSCSICSProviderModule OK
HPUXSCSIProviderModule OK
SFMProviderModule OK <--------------------- Must be present
HPUXFCCSProviderModule OK
HPUXFCIndicationProviderModule OK
HPUXFCProviderModule OK
FSProviderModule OK
HPUXIOTreeIndicationProviderModule OK
HPUXLANProviderModule OK
HPUXLANCSProviderModule OK
HPUXLANIndicationProviderModule OK
HP_iCODProviderModule OK
HP_iCAPProviderModule OK
HP_GiCAPProviderModule OK
HP_VParProviderModule OK
HP_UtilizationProviderModule OK
IOTreeModule OK
# cimprovider -l -m SFMProviderModule
CPUProvider
CPUStatusProvider
EMSWrapperProvider
EventIndicationConsumer
MemoryProvider
MemoryStatusProvider
StateChangeIndicationProvider
ChassisProvider
CoolingStatusProvider
PowerStatusProvider
ThermalProvider
VoltageProvider
EMDProvider
EMArchiveConsumer
EMEmailConsumer
FMDProvider
HealthStateProvider
HPUX_ControlProvider
EvmCimProvider
If the provider is not registered properly, enter the following command at the HP-UX prompt:
# cimprovider -ls | grep SFMProviderModule
If the following output is displayed, all the providers are registered properly:
SFMProviderModule OK
If the output displayed is different from this output, the provider module is not registered. To register the provider module, enter the following command at the HP-UX prompt:
# cimmof -n root/PG_InterOp /opt/sfm/schemas/mof/SFMProvidersR.mof
For the EMSHAProvider Module use the following command to register it:
# /opt/wbem/bin/cimmof -w -n root/PG_InterOp /etc/opt/resmon/mof/EMSHAProviderR.mof
If no errors are displayed, the provider module is registered successfully. If errors are displayed, restart the CIMOM by entering the following command at the HP-UX prompt:
# cimserver
After the CIMOM restarts, enter the following command at the HP-UX prompt to register the provider module:
# cimmof -n root/PG_InterOp /opt/sfm/schemas/mof/SFMProvidersR.mof # /opt/wbem/bin/cimmof -w -n root/PG_InterOp /etc/opt/resmon/mof/EMSHAProviderR.mof
After the provider module is registered, create a link between the SFM providers and the CIMOM by entering the following command at the HP-UX prompt:
On Itanium-based systems, enter:
#ln -s /opt/sfm/lib/libsfmproviders.1 \ /opt/wbem/providers/lib/libsfmproviders.so
On PA-RISC-based systems, enter:
# ln -s /opt/sfm/lib/libsfmproviders.1 \ /opt/wbem/providers/lib/libsfmproviders.sl
F) If using a user other than root for WBEM subscribtions, check:
-- The user exists, e.g. wbem user:
#pwget -n wbem wbem:cKPlOKuqC72Tg:114:20::/home/wbem:/sbin/sh
-- If user hasn't been added, add it:
# useradd -m wbem # passwd wbem
-- Check the planned CIM configuration and make sure the following variables are set to true:
enableSubscriptionsForNonprivilegedUsers enableNamespaceAuthorization
# cimconfig -lp
sslClientVerificationMode=disabled
enableSubscriptionsForNonprivilegedUsers=false
shutdownTimeout=30
authorizedUserGroups=
enableRemotePrivilegedUserAccess=false
enableHttpsConnection=true
enableHttpConnection=true
# cimconfig -s enableSubscriptionsForNonprivilegedUsers=true -p
# cimconfig -s enableNamespaceAuthorization=true -p
# cimserver -s
# cimserver
Verify cimserver started and is working:
# tail /var/adm/syslog/syslog.log | grep cimserver
Mar 20 11:09:20 system cimserver[5267]: PGS10031: CIM server HP-UX WBEM Services stopped.
Mar 20 11:09:20 system cimserver[5266]: exited normally
Mar 20 11:09:33 system cimserver[20785]: starting
Mar 20 11:09:40 system cimserver[20788]: PGS10026: The CIM server is listening on HTTPS port 5,989.
Mar 20 11:09:40 system cimserver[20788]: PGS10028: The CIM server is listening on the local connection socket.
Mar 20 11:09:40 system cimserver[20788]: PGS10030: Started HP-UX WBEM Services version A.02.07.02.
# cimconfig -lc
sslClientVerificationMode=disabled
enableSubscriptionsForNonprivilegedUsers=true <--- set to true
shutdownTimeout=30
authorizedUserGroups=
enableRemoteprivilegedUserAccess=true
enableHttpsConnection=true
enableNamespaceAuthroization=true <--- set to true
enablehttpConnection=false
-- Check the available namespaces under this directory /var/opt/wbem/repository
# ls /var/opt/wbem/repository
root root#cimv2 root#cimv2#npar
root#PG_InterOp root#cimv2#hpvm root#cimv2#vpar
root#PG_Internal root#cimv2#hpvm#pending
-- Add read and write authorizations for the user ‘wbem’ to each of the namespaces found under /var/opt/wbem/repository
#cimauth –a –u wbem –n root/cimv2 –R –W PGC00407: Authorizations added successfully. #cimauth –a –u wbem –n root/PG_InterOp –R –W PGC00407: Authorizations added successfully. #cimauth –a –u wbem –n root/PG_Internal –R –W PGC00407: Authorizations added successfully. #cimauth –a –u wbem –n root/cimv2/npar –R –W PGC00407: Authorizations added successfully. #cimauth –a –u wbem –n root/cimv2/vpar –R –W PGC00407: Authorizations added successfully. #cimauth –a –u wbem –n root/cimv2/hpvm –R -W
--Verify the user's authorization
# cimauth -l wbem, root/cimv2, "rw" wbem, root/PG_InterOp, "rw" wbem, root/PG_Internal, "rw" wbem, root/cimv2/npar, "rw" wbem, root/cimv2/vpar, "rw"
G) Verify the WBEM https port is listening:
# netstat -a | grep wbem tcp 0 0 *.wbem-https *.* LISTEN tcp 0 0 *.wbem-https *.* LISTEN
If that fails check the default port numbers:
# grep wbem /etc/services wbem-http 5988/tcp # Web-Based Enterprise Management HTTP wbem-http 5988/udp # Web-Based Enterprise Management HTTP wbem-https 5989/tcp # Web-Based Enterprise Management HTTPS wbem-https 5989/udp # Web-Based Enterprise Management HTTPS
# netstat -an |grep 5989 tcp 0 0 *.5989 *.* LISTEN tcp 0 0 *.5989 *.* LISTEN
H) Verify the connectivity between the managed node and the SIM server:
#ping <simserver hostname> #ping <simserver ip> # nsquery hosts <simserver hostname> # nsquery hosts <simserver ip>
I) Check that there are subscriptions listed externally with the SIM server (sim server should be listed in here):
# /opt/sfm/bin/evweb subscribe -L -b external
Filter Name Handler Name Query Destinat Destination Url
================ =============== ======================================== ======== ==============================
HPSIM_barbados_0 HPSIM_barbados select * from HP_DeviceIndication UNKNOWN
HPSIM_barbados_1 HPSIM_barbados select * from HP_ThresholdIndication UNKNOWN
HPSIM_froghair_0 HPSIM_froghair select * from HP_DeviceIndication UNKNOWN
HPSIM_froghair_1 HPSIM_froghair select * from HP_ThresholdIndication UNKNOWN
J) Check that the diagnostic mode is set to SFM:
# /opt/sfm/bin/sfmconfig -w –q EMS hardware monitors are disabled & SysFaultMgmt is monitoring devices.
To switch to SFM Mode:
# /opt/sfm/bin/sfmconfig -w –s
K) Send a event using the native provider (11.23 and after):
# /opt/sfm/bin/sfmconfig -t -a Sending test event for fpl_em monitor. Sending test event for ia64_corehw monitor. Sending test event for memory monitor. Sending test event for processor monitor. Sending test event for backbone monitor.
Send a test event using the wrapper provider (for systems running 11.11):
# /etc/opt/resmon/lbin/send_test_event –v <monitor name> E.g.: # /etc/opt/resmon/lbin/send_test_event –v disk_em
Check that the event is listed in evweb:
# evweb eventviewer -L
EvArchNo Severity Event # Event Category Archive Time Summary
========= =========== ============== ============== ============== ==============
5 Information 103 Processor 2008-07-20 14: This is a t...
4 Information 103 System Hard... 2008-07-19 18: This is a t...
3 Information 103 Processor 2008-07-19 18: This is a t...
2 Information 103 Memory 2008-07-19 18: This is a t...
1 Information 103 System Hard... 2008-07-19 18: This is a t...
To Launch EVWEB for viewing event subscriptions using the HP SMH GUI, complete the following steps:
- Log in to the System Management Homepage.
- To log in to HP SMH, enter http://<hostname>: 2301 in the address bar of the Web browser. The HP SMH login screen is displayed.
- Enter your user name and password in the appropriate text boxes.
- Click Sign In on the login screen.
- The HP SMH home page is displayed.
- Select Logs in the main menu.
- The Logs page is displayed. This page displays various applications that are plugged into HP SMH.
- Select Event Viewer in Evweb.
- The Event Viewer page is displayed.
You can view and delete WBEM indications using the Event Viewer.
L) Check whether the HP-Defined filters are enabled:
# /opt/sfm/bin/sfmconfig -m list -t ALL
Filter Name : General Filter
Filter Type : HP Defined Filter
Filter Unique Identifier : 1
Filter Query : Select * from HP_DeviceIndication
Filter Query Language : WQL
Filter Source Namespace : root/cimv2
Filter Description : General Device Indications.
Filter State : Enabled Filter State
Filter Last Operation : No Operation
========================================================
If need to enable them use the command:
#/opt/sfm/bin/sfmconfig –m enable – n “General Filter” –u 1 –t HP
Filter Name : General Filter
Filter Type : HP Defined Filter
Filter Unique Identifier : 1
Filter Query : Select * from HP_DeviceIndication
Filter Query Language : WQL
Filter Source Namespace : root/cimv2
Filter Description : General Device Indications.
Filter State : Enabled Filter State
Filter Last Operation : Enable Filter
========================================================
M) List and display information about all or selected indication subscriptions:
- cimsub -ls
N) If necessary run these steps:
# swconfig -u SysFaultMgmt # /sbin/init.d/sfm stop # ps -ef | grep post # cd /var/opt/sfmdb # rm -r pgsql # swconfig SysFaultMgmt # /sbin/init.d/sfm start
On the CMS/SIM Server
Note: If the SIM server is Windows all commands should be found in C:\Program Files\HP\Systems Insight Manager\bin and if its HPUX they should be located at /opt/mx/bin/
A) Make sure the SIM server is resolving the hostname of the managed node properly and is able to communicate with it:
- ping <managednodehostname>
- ping <managednode ip>
- nslookup <managednodehostname>
- nslookup <managednode ip>
- telnet <managenodeip> 5989 (Make sure the port 5989 is not blocked)
A) Verify the managed node has been discovered:
# mxnode -lt | grep system system system.hp.com HP-UX B.11.23 U Mar 20, 2009 12:41:08 PM ip_address
If it hasn't been discovered, then add it to the SIM server database:
# mxnode -a <managednode> -w Waiting for node(s) to be added... Node has been added/modified: <managednode>
This command may take some time to complete. Try to use always the -w option, this ensures that the node is added to the SIM Server database completely.
B) Check which protocols are registered against the managed node
# mxnode -ld system | grep -i protocol Protocols: HTTP: SMH:2.0 SNMP:1.0 SSH:SSH-2.0-OpenSSH_5.1p1+sftpfilecontrol-v1.2-hpn13v5
If the protocol is not listed, check the credentials and remove/add credentials as needed:
# mxnodesecurity -l -n <managednode>
# mxnodesecurity -r -p wbem -n <managednode>
Node/system credential removed: <managednode>
# mxnodesecurity -a -p wbem -c wbem:password -n <managednode>
Node/system credential added: <managednode>
# mxnodesecurity -l -n <managednode>
Listing all system credentials...
NODENAME PROTOCOL USERNAME PASSWORD TRYOTHERS
...
managednode wbem wbem ******** No
# mxnode -ld system | grep -i protocol
Protocols: HTTP: WBEM 1.0:SMH:2.0 SNMP:1.0 SSH:SSH-2.0-OpenSSH_5.1p1+sftpfilecontrol-v1.2-hpn13v5
Readd/Reidentify the system
# mxnode -a <managednode> -w
Waiting for node(s) to be added...
Node has been added/modified: <managednode>
If the protocol is still not listed, make sure that under Global Protocol Settings WBEM is enabled. After enabling the WBEM protocol, then reidentify the system.
You can use also in SIM server Options ---> Identify System and check if any other error is reported.
C) Subscribe the managed node to wbem events:
#mxwbemsub -a -n <managednode> Create Subscriptions for: <managednode> Indication subscription successfully created.
If this fails, run vseassist for the system on the CMS (this is only if SIM server is HPUX):
#vseassist -l -n system
D) If necessary grant full access to root user on the SIM server:
#mxuser -m root -p full
Also you can debug the user as
#mxuser -m root -p debug
Reference
- http://www.docs.hp.com/en/5992-4854/ch06s02.html
- WBEM Providers
- How to troubleshoot WBEM subscriptions?
- How to rebuild the /var/opt/wbem/repository directory for WBEM Services?
- How to recreate WBEM namespaces under /var/opt/wbem/repository?
- How to recreate WBEM/SMH Certificates?
- emr_na-c01919042-1
