Creating the 11.2 OEM (with Oracle Flex) with emca Fails with Message: WARNING: Error determining the nodes associated with cluster database for ASM

Hi,

When I try to create Oracle Enterprise Manager with EMCA for 11.2.0.3 database running on 18c Grid Infrastructure, it fails with following error mesaage

Database Control (DBC) 11.2 web site creation with emca fails for Oracle Flex with this error message:

WARNING: Error determining the nodes associated with cluster database for ASM
Apr 9, 2019 11:28:18 PM oracle.sysman.emcp.ParamsManager queryDBForASMNodeList
CONFIG: Stack trace:
PRCA-1018 : Failed to retrieve ASM instance list for ASM asm
PRCR-1115 : Failed to find entities of type resource instance that match filters ((TYPE == ora.asm.type) && (NAME == ora.asm)) and contain attributes LAST_SERVER,USR_ORA_INST_NAME
CRS-2653: Special value evaluation must be associated with a resource instance
at oracle.cluster.impl.asm.ASMImpl.getInstances(ASMImpl.java:787)
at oracle.cluster.impl.asm.ASMImpl.allInstances(ASMImpl.java:702)
at oracle.sysman.emcp.ParamsManager.queryDBForASMNodeList(ParamsManager.java:1899)
at oracle.sysman.emcp.ParamsManager.getASMNodeList(ParamsManager.java:1673)
at oracle.sysman.emcp.ParamsManager.validatePassword(ParamsManager.java:3483)
at oracle.sysman.emcp.EMConfigAssistant.promptForData(EMConfigAssistant.java:710)
at oracle.sysman.emcp.EMConfigAssistant.promptForParams(EMConfigAssistant.java:3028)
at oracle.sysman.emcp.EMConfigAssistant.displayWarnsAndPromptParams(EMConfigAssistant.java:3065)
at oracle.sysman.emcp.EMConfigAssistant.getDisplayAndPromptWarnsParms(EMConfigAssistant.java:3090)
at oracle.sysman.emcp.EMConfigAssistant.performConfiguration(EMConfigAssistant.java:1116)
at oracle.sysman.emcp.EMConfigAssistant.statusMain(EMConfigAssistant.java:583)
at oracle.sysman.emcp.EMConfigAssistant.main(EMConfigAssistant.java:529)
Caused by: PRCR-1115 : Failed to find entities of type resource instance that match filters ((TYPE == ora.asm.type) && (NAME == ora.asm)) and contain attributes LAST_SERVER,USR_ORA_INST_NAME
CRS-2653: Special value evaluation must be associated with a resource instance
at oracle.cluster.impl.crs.CRSNative.searchEntities(CRSNative.java:1016)
at oracle.cluster.impl.crs.CRSFactoryImpl.searchEntities(CRSFactoryImpl.java:637)
at oracle.cluster.impl.asm.ASMImpl.getInstances(ASMImpl.java:768)
... 11 more
Caused by: CRS-2653: Special value evaluation must be associated with a resource instance
at oracle.cluster.impl.crs.CRSNativeResult.addComp(CRSNativeResult.java:171)
at oracle.cluster.impl.crs.CRSNative.doQueryEntities(Native Method)
at oracle.cluster.impl.crs.CRSNative.searchEntities(CRSNative.java:1006)
... 13 more
Apr 9, 2019 11:28:18 PM oracle.sysman.emcp.ParamsManager getInaccessibleSidList
CONFIG: Invalid Node/Sid List specified.

This is due to bug number 20416112.

In the Oracle Flex ASM configuration not all ASM instances are required to be Up or even to have been yet started. In such a case, the DBC web site configuration process can fail.

Querying the ASM instances for activity shows the following, namely one OFFLINE ASM instance.

orcnode1 /home/oracle> crsctl stat res ora.asm -t 
--------------------------------------------------------------------------
Name Target State Server State details 
--------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE orcnode1 Started,STABLE
2 ONLINE ONLINE orcnode2 Started,STABLE
3 OFFLINE OFFLINE STABLE
--------------------------------------------------------------------------

Please also check clsuter mode.

orcnode1 /home/oracle> asmcmd showclustermode
ASM cluster : Flex mode enabled – Direct Storage Access
orcnode1 /home/oracle> srvctl config asm
ASM home: <CRS home>
Password file: +DATA/orapwASM
Backup of Password file:
ASM listener: LISTENER_REP
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM,ASMNET2LSNR_ASM

For workaround, please issue this command with grid os user.
orcnode1 /home/oracle> srvctl modify asm -count ALL

And again check status.

orcnode1 /home/oracle> srvctl config asm
ASM home: <CRS home>
Password file: +DATA/orapwASM
Backup of Password file:
ASM listener: LISTENER_REP
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM,ASMNET2LSNR_ASM

orcnode1 /home/oracle> crsctl stat res ora.asm -t
--------------------------------------------------------------------------
Name Target State Server State details 
--------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE orcnode1 Started,STABLE
2 ONLINE ONLINE orcnode2 Started,STABLE
--------------------------------------------------------------------------
orcnode1 /home/oracle>

This action will decrease the number of ASM instances expected by emca to the number of ONLINE instances returned by  the “crsctl stat res ora.asm -t” command and allow the configuration to continue.

Now, you can retry to run EMCA with cluster parameter.

Advertisements

CRS-2317:Fatal error: cannot get local GPnP security keys (wallet) messages reported in instance alert log

Hi,

After installation Oracle 18c RAC database, I got following error messages in alert log file. Database wass using public ip for the cluster_interconnects instead of HAIP.From.

Alert log file content

[USER(4431)]CRS-2317: Fatal error: cannot get local GPnP security keys (wallet).
[USER(4431)]CRS-2316:Fatal error: cannot initialize GPnP, CLSGPNP_ERR (Generic GPnP error).
kggpnpInit: failed to init gpnp
WARNING: No cluster interconnect has been specified. Depending on
the communication driver configured Oracle cluster traffic
may be directed to the public interface of this machine.
Oracle recommends that RAC clustered databases be configured
with a private interconnect for enhanced security and
performance.
if you check gv$cluster_interconnects view, it returns no rows.
If you are not using Clusterware (Single instance) than you can ignore this messages. But if you are using RAC like my case then workaround is please set database init parameter for cluster interconnects.alter system set cluster_interconnects=’10.1.241.1:10.1.231.1′ scope=both sid=’orcl1′

alter system set cluster_interconnects=’10.1.241.2:10.1.231.2′ scope=both sid=’orcl2′

Restart database and recheck your instance alert logs.

Also check gv$cluster_interconnects

select * from gv$cluster_interconnects;

INST_ID|NAME |IP_ADDRESS |IS_|SOURCE
———-|—————|—————-|—|——————————-
2|eno2 |10.1.231.2 |NO |cluster_interconnects parameter
2|eno3 |10.1.241.2 |NO |cluster_interconnects parameter
1|eno2 |10.1.231.1 |NO |cluster_interconnects parameter
1|eno3 |10.1.241.1 |NO |cluster_interconnects parameter

For example, if you are running two instances of Oracle for two databases on the same machine, then you can load balance the interconnect traffic to different physical interconnects. This does not reduce Oracle availability.

CLUSTER_INTERCONNECTS can be used in Oracle Real Application Clusters environments to indicate cluster interconnects available for use for the database traffic. Use this parameter if you need to override the default interconnect configured for the database traffic, which is stored in the cluster registry. This procedure also may be useful with Data Warehouse systems that have reduced availability requirements and high interconnect bandwidth demands.

How To Modify SCAN Name or SCAN VIP Addresses

Hi,

For some reasons you may want to change scan name or scan ip addresses.

As grid user stop resources:

$GRID_HOME/bin/srvctl stop scan_listener
$GRID_HOME/bin/srvctl stop scan
Update DNS entries for new scan name and ip addresses.

As root user modify SCAN:

$GRID_HOME/bin/srvctl modify scan -n <new_scan_name>

For 11.2.0.1 only, if you intend to change SCAN name, due to bug 9603829,  you must execute the following:

$GRID_HOME/bin/crsctl modify type ora.scan_vip.type -attr “ATTRIBUTE=SCAN_NAME,DEFAULT_VALUE=<new_scan_name>”

After SCAN name was changed, update database spfile parameter for remote_listener to the new scan.

As grid user modify and start resources:

$GRID_HOME/bin/srvctl modify scan_listener -u
$GRID_HOME/bin/srvctl start scan_listener

Please confirm the change

$GRID_HOME/bin/srvctl config scan
$GRID_HOME/bin/srvctl config scan_listener

How To Modify RAC SCAN Listener Port

Hi,

For some security reasons, you may want to change RAC scan listener port number.

As grid user:

Modify SCAN listener port

$GRID_HOME/bin/srvctl modify scan_listener -p <new_scan_port>

Please don’t forget to update database init.ora/spfile parameter file as:

alter system set remote_listener='<scan_name>:<new_port_number>’ scope=both sid=’*’;

Restart SCAN listener so the new port will be effect

$GRID_HOME/bin/srvctl stop scan_listener
$GRID_HOME/bin/srvctl start scan_listener

Confirm it

$GRID_HOME/bin/srvctl config scan_listener

Changing Public Network Interface, Subnet or Netmask in Oracle Clusterware

Hi,

If the Public Network interface change involves different subnet(netmask) or interface, delete the existing interface information from OCR and add it back with the correct information is required.  In the example here, the subnet is changed from 10.2.156.0  to 10.2.166.0

% $GRID_HOME/bin/oifcfg delif -global eth0/10.2.156.0
% $GRID_HOME/bin/oifcfg setif -global eth0/10.2.166.0:public

Then make the change at OS layer. There is no requirement to restart Oracle clusterware unless OS change requires a node reboot. This can be done in rolling fashion.

Once public network is changed, its associated VIP and SCAN VIP are also required to change.

Please ensure that public network changes are made first. If there is a node reboot or Clusterware restart after the OS network change, vip will not start.

Gather Current VIP Configuration

as Grid Infrastructure owner:

$ srvctl config nodeapps -a

Verify VIP status. It should show VIPs are ONLINE

$ crsctl stat res -t

$ ifconfig -a

– VIP logical interface is bound to the public network interface

Stopping Resources

Stop the nodeapps resources (and all dependent resources ASM/DB only if required):

as Grid Infrastructure owner:

$ srvctl stop instance -d ORCDB -n orcnode1
$ srvctl stop vip -n orcnode1 -f

Verify VIP is now OFFLINE and the interface is no longer bound to the public network

Modifying VIP and Its Associated Attributes

Determine the new VIP IP/subnet/netmask or VIP hostname, make the network change on OS first, ensure the new VIP is registered in DNS or modified in /etc/hosts . If the network interface is changed, ensure the new interface is available on the server before proceeding with the modification.

For example:
New VIP is: 10.2.166.21 orcnode1-nvip
new subnet is 10.2.166.0
new netmask is 255.255.255.0
new interface is eth2

Modify the VIP resource, as root user:

$ srvctl modify nodeapps -n orcnode1 -A orcnode1-nvip/255.255.255.0/eth2

Verify the change

$ srvctl config nodeapps -n orcnode1 -a
VIP exists.: /orcnode1-nvip/10.2.166.21/255.255.255.0/eth2

Start the nodeapps and the other resources

as Grid Infrastructure owner:

$ srvctl start vip -n orcnode1
$ srvctl start listener -n orcnode1
$ srvctl start instance -d ORCDB -n orcnode1

Verify the new VIP is ONLINE and bind to the public network interface

$ crsctl stat res -t
$ ifconfig -a

Repeat the same steps for the rest nodes in the cluster only if the similar change is required.