Shut Down, Reset, or Power Cycle the Oracle MiniCluster S7-2 Engineered System

Hi,

In this post, I want to share the required steps for Startup, Shut Down, Reset, or Power Cycle the Oracle MiniCluster S7-2 Engineered System.

Notify affected users that the server will be shut down.
Log into one of the nodes as an MCMU administrator such as mcinstall

mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu status -G -k

List a Summary of All DB VM Groups

mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu tenant -G -l

Stop the DB grid infrastructure

mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu stop -G -n VMgroupname

Here, where VMgroupname is the name of the DB VM group. You can learn with “mcmu tenant -G -l”

Stop the GI in the kernel zones on node1 and node2.

mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu stop -G -k node1
mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu stop -G -k node2

Re-check Workgroup status

mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu status -G -k

Shut down all the zones in VM group dpwyadm

mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu stop -V -n dpwyadm

Check running status

mcinstall@dpwadm-n1:/var/home/mcinstall % zoneadm list -civ

If there are “running” state then stop the kernel zones.

mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu stop -Z -k node1
[INFO ] Log file path : dpwadm-n1:/var/opt/oracle.minicluster/setup/logs/mcmu_121818_131505.log
[INFO ] Check if initial setup complete
[INFO ] Checking if nodes are in sync
Do you want to shut down zone acfskz? [yes/no] (no): yes
[INFO ] Zone acfskz has been shut down
mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu stop -Z -k node2
[INFO ] Log file path : dpwadm-n1:/var/opt/oracle.minicluster/setup/logs/mcmu_121818_131716.log
[INFO ] Check if initial setup complete
[INFO ] Checking if nodes are in sync
Do you want to shut down zone acfskz? [yes/no] (no): yes
[INFO ] Zone acfskz has been shut down

OR You want to shut down the whole system with one command.

mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu stop -S

Shutdown one node:

mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu stop -N node1
mcinstall@dpwadm-n1:/var/home/mcinstall % mcmu stop -N node2

For a full power down, perform the remaining steps. Now, you can stop ILOMs. Oracle ILOM CLI – In a terminal window, enter:

Oracle(R) Integrated Lights Out Manager

Version 4.0.2.2.c r124557

Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: dpwadm-n1-ilom

-> stop /System/
Are you sure you want to stop /System (y/n)? y
Stopping /System

For Starting the whole system, Log in to ILOMs

Oracle(R) Integrated Lights Out Manager

Version 4.0.2.2.c r124557

Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: dpwadm-n1-ilom

-> start /System

Check System

-> show /System/

/System
Targets:
Open_Problems (0)
Processors
Memory
Power
Cooling
Storage
Networking
PCI_Devices
Firmware
Log

Properties:
health = OK
health_details = –
open_problems_count = 0
type = Rack Mount
model = MiniCluster S7-2
qpart_id = XXXXXXXX
part_number = MiniCluster S7-2
serial_number = XXXXXXXXXX
component_model = SPARC S7-2
component_part_number = XXXXXXX
component_serial_number = XXXXXXXX
system_identifier = Oracle MiniCluster S7-2 XXXXXXXXXX
system_fw_version = Sun System Firmware 9.8.5.c 2018/05/03 17:22
primary_operating_system = Oracle Solaris 11.3 SPARC
primary_operating_system_detail = –
host_primary_mac_address = 00:10:e0:c0:bf:30
ilom_address = 10.102.106.141
ilom_mac_address = 00:10:E0:C0:BF:39
locator_indicator = Off
power_state = Off
actual_power_consumption = 471 watts
action = (Cannot show property)

Commands:
cd
reset
set
show
start
stop

->

Also please check for faults.

Oracle(R) Integrated Lights Out Manager

Version 4.0.2.2.c r124557

Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: dpwadm-n1-ilom

-> start /SP/faultmgmt/shell/
Are you sure you want to start /SP/faultmgmt/shell (y/n)? y

faultmgmtsp>
faultmgmtsp> fmadm faulty
No faults found
faultmgmtsp> exit

Now, log into one of the nodes as an MCMU administrator such as mcinstall and check the system

zoneadm list -civ

mcmu status -G -k

Advertisements

Creating the 11.2 OEM (with Oracle Flex) with emca Fails with Message: WARNING: Error determining the nodes associated with cluster database for ASM

Hi,

When I try to create Oracle Enterprise Manager with EMCA for 11.2.0.3 database running on 18c Grid Infrastructure, it fails with following error mesaage

Database Control (DBC) 11.2 web site creation with emca fails for Oracle Flex with this error message:

WARNING: Error determining the nodes associated with cluster database for ASM
Apr 9, 2019 11:28:18 PM oracle.sysman.emcp.ParamsManager queryDBForASMNodeList
CONFIG: Stack trace:
PRCA-1018 : Failed to retrieve ASM instance list for ASM asm
PRCR-1115 : Failed to find entities of type resource instance that match filters ((TYPE == ora.asm.type) && (NAME == ora.asm)) and contain attributes LAST_SERVER,USR_ORA_INST_NAME
CRS-2653: Special value evaluation must be associated with a resource instance
at oracle.cluster.impl.asm.ASMImpl.getInstances(ASMImpl.java:787)
at oracle.cluster.impl.asm.ASMImpl.allInstances(ASMImpl.java:702)
at oracle.sysman.emcp.ParamsManager.queryDBForASMNodeList(ParamsManager.java:1899)
at oracle.sysman.emcp.ParamsManager.getASMNodeList(ParamsManager.java:1673)
at oracle.sysman.emcp.ParamsManager.validatePassword(ParamsManager.java:3483)
at oracle.sysman.emcp.EMConfigAssistant.promptForData(EMConfigAssistant.java:710)
at oracle.sysman.emcp.EMConfigAssistant.promptForParams(EMConfigAssistant.java:3028)
at oracle.sysman.emcp.EMConfigAssistant.displayWarnsAndPromptParams(EMConfigAssistant.java:3065)
at oracle.sysman.emcp.EMConfigAssistant.getDisplayAndPromptWarnsParms(EMConfigAssistant.java:3090)
at oracle.sysman.emcp.EMConfigAssistant.performConfiguration(EMConfigAssistant.java:1116)
at oracle.sysman.emcp.EMConfigAssistant.statusMain(EMConfigAssistant.java:583)
at oracle.sysman.emcp.EMConfigAssistant.main(EMConfigAssistant.java:529)
Caused by: PRCR-1115 : Failed to find entities of type resource instance that match filters ((TYPE == ora.asm.type) && (NAME == ora.asm)) and contain attributes LAST_SERVER,USR_ORA_INST_NAME
CRS-2653: Special value evaluation must be associated with a resource instance
at oracle.cluster.impl.crs.CRSNative.searchEntities(CRSNative.java:1016)
at oracle.cluster.impl.crs.CRSFactoryImpl.searchEntities(CRSFactoryImpl.java:637)
at oracle.cluster.impl.asm.ASMImpl.getInstances(ASMImpl.java:768)
... 11 more
Caused by: CRS-2653: Special value evaluation must be associated with a resource instance
at oracle.cluster.impl.crs.CRSNativeResult.addComp(CRSNativeResult.java:171)
at oracle.cluster.impl.crs.CRSNative.doQueryEntities(Native Method)
at oracle.cluster.impl.crs.CRSNative.searchEntities(CRSNative.java:1006)
... 13 more
Apr 9, 2019 11:28:18 PM oracle.sysman.emcp.ParamsManager getInaccessibleSidList
CONFIG: Invalid Node/Sid List specified.

This is due to bug number 20416112.

In the Oracle Flex ASM configuration not all ASM instances are required to be Up or even to have been yet started. In such a case, the DBC web site configuration process can fail.

Querying the ASM instances for activity shows the following, namely one OFFLINE ASM instance.

orcnode1 /home/oracle> crsctl stat res ora.asm -t 
--------------------------------------------------------------------------
Name Target State Server State details 
--------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE orcnode1 Started,STABLE
2 ONLINE ONLINE orcnode2 Started,STABLE
3 OFFLINE OFFLINE STABLE
--------------------------------------------------------------------------

Please also check clsuter mode.

orcnode1 /home/oracle> asmcmd showclustermode
ASM cluster : Flex mode enabled – Direct Storage Access
orcnode1 /home/oracle> srvctl config asm
ASM home: <CRS home>
Password file: +DATA/orapwASM
Backup of Password file:
ASM listener: LISTENER_REP
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM,ASMNET2LSNR_ASM

For workaround, please issue this command with grid os user.
orcnode1 /home/oracle> srvctl modify asm -count ALL

And again check status.

orcnode1 /home/oracle> srvctl config asm
ASM home: <CRS home>
Password file: +DATA/orapwASM
Backup of Password file:
ASM listener: LISTENER_REP
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM,ASMNET2LSNR_ASM

orcnode1 /home/oracle> crsctl stat res ora.asm -t
--------------------------------------------------------------------------
Name Target State Server State details 
--------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE orcnode1 Started,STABLE
2 ONLINE ONLINE orcnode2 Started,STABLE
--------------------------------------------------------------------------
orcnode1 /home/oracle>

This action will decrease the number of ASM instances expected by emca to the number of ONLINE instances returned by  the “crsctl stat res ora.asm -t” command and allow the configuration to continue.

Now, you can retry to run EMCA with cluster parameter.

CRS-2317:Fatal error: cannot get local GPnP security keys (wallet) messages reported in instance alert log

Hi,

After installation Oracle 18c RAC database, I got following error messages in alert log file. Database wass using public ip for the cluster_interconnects instead of HAIP.From.

Alert log file content

[USER(4431)]CRS-2317: Fatal error: cannot get local GPnP security keys (wallet).
[USER(4431)]CRS-2316:Fatal error: cannot initialize GPnP, CLSGPNP_ERR (Generic GPnP error).
kggpnpInit: failed to init gpnp
WARNING: No cluster interconnect has been specified. Depending on
the communication driver configured Oracle cluster traffic
may be directed to the public interface of this machine.
Oracle recommends that RAC clustered databases be configured
with a private interconnect for enhanced security and
performance.
if you check gv$cluster_interconnects view, it returns no rows.
If you are not using Clusterware (Single instance) than you can ignore this messages. But if you are using RAC like my case then workaround is please set database init parameter for cluster interconnects.alter system set cluster_interconnects=’10.1.241.1:10.1.231.1′ scope=both sid=’orcl1′

alter system set cluster_interconnects=’10.1.241.2:10.1.231.2′ scope=both sid=’orcl2′

Restart database and recheck your instance alert logs.

Also check gv$cluster_interconnects

select * from gv$cluster_interconnects;

INST_ID|NAME |IP_ADDRESS |IS_|SOURCE
———-|—————|—————-|—|——————————-
2|eno2 |10.1.231.2 |NO |cluster_interconnects parameter
2|eno3 |10.1.241.2 |NO |cluster_interconnects parameter
1|eno2 |10.1.231.1 |NO |cluster_interconnects parameter
1|eno3 |10.1.241.1 |NO |cluster_interconnects parameter

For example, if you are running two instances of Oracle for two databases on the same machine, then you can load balance the interconnect traffic to different physical interconnects. This does not reduce Oracle availability.

CLUSTER_INTERCONNECTS can be used in Oracle Real Application Clusters environments to indicate cluster interconnects available for use for the database traffic. Use this parameter if you need to override the default interconnect configured for the database traffic, which is stored in the cluster registry. This procedure also may be useful with Data Warehouse systems that have reduced availability requirements and high interconnect bandwidth demands.

Direct NFS: Failed to set socket buffer size.wtmax=[1048576] rtmax=[1048576], errno=-1

Hi,

Today, I faced the below errors in my alert log file.

Direct NFS: Failed to set socket buffer size.wtmax=[1056768] rtmax=[1056768], errno=-1
Mon Apr 08 20:00:51 2019
Direct NFS: Failed to set socket buffer size.wtmax=[1056768] rtmax=[1056768], errno=-1
Mon Apr 08 20:00:51 2019
Direct NFS: Failed to set socket buffer size.wtmax=[1056768] rtmax=[1056768], errno=-1
Mon Apr 08 20:00:51 2019
Direct NFS: Failed to set socket buffer size.wtmax=[1056768] rtmax=[1056768], errno=-1
Mon Apr 08 20:00:51 2019
Direct NFS: Failed to set socket buffer size.wtmax=[1056768] rtmax=[1056768], errno=-1
Direct NFS: Failed to set socket buffer size.wtmax=[1056768] rtmax=[1056768], errno=-1
Mon Apr 08 20:00:51 2019
Direct NFS: Failed to set socket buffer size.wtmax=[1056768] rtmax=[1056768], errno=-1

These errors are written to my alert log file during backup operation to NFS mounted backup store. The root cause is tcp_max_buf  is set too small. The error can be fixed by increasing the value of tcp_max_buf.

To display its current value, please run:

# /usr/sbin/ndd /dev/tcp tcp_max_buf
1048576

And increase as following

# /usr/sbin/ndd -set /dev/tcp tcp_max_buf 1056768

How To Modify SCAN Name or SCAN VIP Addresses

Hi,

For some reasons you may want to change scan name or scan ip addresses.

As grid user stop resources:

$GRID_HOME/bin/srvctl stop scan_listener
$GRID_HOME/bin/srvctl stop scan
Update DNS entries for new scan name and ip addresses.

As root user modify SCAN:

$GRID_HOME/bin/srvctl modify scan -n <new_scan_name>

For 11.2.0.1 only, if you intend to change SCAN name, due to bug 9603829,  you must execute the following:

$GRID_HOME/bin/crsctl modify type ora.scan_vip.type -attr “ATTRIBUTE=SCAN_NAME,DEFAULT_VALUE=<new_scan_name>”

After SCAN name was changed, update database spfile parameter for remote_listener to the new scan.

As grid user modify and start resources:

$GRID_HOME/bin/srvctl modify scan_listener -u
$GRID_HOME/bin/srvctl start scan_listener

Please confirm the change

$GRID_HOME/bin/srvctl config scan
$GRID_HOME/bin/srvctl config scan_listener