Revive RMAN: Mastering the Art of Resuming Failed Duplicate Operations

 Hey Oracle DBAs! It's been a while, Oracle community! But I'm back with a helpful post. :-)

Today, I faced an issue while performing an RMAN duplicate to build out a standby database (Oracle version 19.25) on an ExaCS VM. The database is 110 TB in size. I was doing an active duplicate over the WAN for 2 days. Everything was going smoothly until it reached ~94% completion... and then it FAILED!!

I didn't want to risk restarting the entire RMAN duplicate process again out of concern that the same issue would occur. Here's my error:



RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 04/10/2025 09:57:48
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-19875: multisection restore or copy not complete for file <datafile_name>

So I started working on getting this completed. After researching, I found that MOS has a good note on getting things going; however, it was intensive. With that being said, I was about to pivot to a different solution, which involved restoring a full backup from OCI storage. But before doing so, I decided to see what would happen if I simply tried to rerun the active duplicate. So here's what I did...

  1. Create a pfile from my spfile for the standby.
  2. Shutdown the standby database instance.
  3. Removed my spfile that was created as part of the RMAN duplicate process.
  4. Startup nomount the standby instance using the pfile I created.
  5. Removed entries referencing spfile from my RMAN duplicate cmd file (see my previous blog posts on standby builds for examples)
  6. Restart the RMAN duplicate.

End result....IT WORKED!!!


Excerpt from the RMAN duplicate logfile...


contents of Memory Script:
{
sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database

Using previous duplicated file +DATAC1/FAKEDB/DATAFILE/system_fake.311.9876543210 for datafile 1 with checkpoint SCN of 14235402086660

Using previous duplicated file +DATAC1/FAKEDB/DATAFILE/sysaux_fake.301.9876543211 for datafile 2 with checkpoint SCN of 14235402113022

Using previous duplicated file +DATAC1/FAKEDB/DATAFILE/fake_data_t2.297.9876543212 for datafile 3 with checkpoint SCN of 14235402115530

Using previous duplicated file +DATAC1/FAKEDB/DATAFILE/users_fake.306.9876543213 for datafile 4 with checkpoint SCN of 14235402117901

Using previous duplicated file +DATAC1/FAKEDB/DATAFILE/fake_dba.305.9876543214 for datafile 5 with checkpoint SCN of 14235402120263

Using previous duplicated file +DATAC1/FAKEDB/DATAFILE/admin_fake_data.304.9876543215 for datafile 6 with checkpoint SCN of 14235402125881

Using previous duplicated file +DATAC1/FAKEDB/DATAFILE/stage_fake_data.291.9876543216 for datafile 7 with checkpoint SCN of 14235402128955

Using previous duplicated file +DATAC1/FAKEDB/DATAFILE/dimension_fake_data.294.9876543217 for datafile 8 with checkpoint SCN of 14235402131767

Using previous duplicated file +DATAC1/FAKEDB/DATAFILE/dimension_fake_idx.296.9876543218 for datafile 9 with checkpoint SCN of 14235402133743
...
...
...
...
set newname for clone datafile 2265 to new;
set newname for clone datafile 2266 to new;
set newname for clone datafile 2267 to new;
set newname for clone datafile 2268 to new;
set newname for clone datafile 2269 to new;
set newname for clone datafile 2270 to new;
set newname for clone datafile 2271 to new;
set newname for clone datafile 2272 to new;
set newname for clone datafile 2273 to new;
set newname for clone datafile 2274 to new;
set newname for clone datafile 2275 to new;
set newname for clone datafile 2276 to new;

restore
from nonsparse section size
20 g from service
'FAKEDB' clone datafile
2198, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222,
2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242,
2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262,
2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276 ;
sql 'alter system archive log current';
}

executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

...
...
...

The highlighted parts were mainly what I was looking for to determine that this actually worked. Not sure if Oracle has simply improved on the RMAN duplicate failure process or if I just got lucky.

Hope this helps anyone/everyone who may be experiencing this issue and thinks that they have to start over from scratch. There's HOPE!!!

Enjoy!


Comments

Post a Comment

Popular posts from this blog

Grid Infrastructure network issue on ODA X5-2

I ran into a situation where client I had supported was having network timeouts and a huge amount of packet drops on their network when trying to connect to their ODA from the application stack.  After many hours (2 days to be exact) of troubleshooting, it was discovered that there was an issue with the Grid Infrastructure network configuration vs what the client's network was configured with. The client's default gateway was set to 10.10.10.1 . However when looking at their network configuration set in the GI, it reflected 10.10.0.0 as this is what Oracle was expecting it to be based off of ipcalc. ipcalc -bnm 10.10.40.42 255.255.0.0 NETMASK=255.255.0.0 BROADCAST=10.10.255.255 NETWORK= 10.10.0.0 ################ [root@oraprd1 .ssh]# oifcfg getif ibbond0  192.168.16.0  global  cluster_interconnect,asm bond0  10.10.0.0  global  public ################ # srvctl config network Network 1 exists Subnet IPv4: 10.10.0.0 /255.255.0.0...
Read more

Upgrade from 11gR2 RAC to 12cR1 RAC using cloning methodology

12.1.0.2 Grid Infrastructure Upgrade: From 11.2.0.4 To 12.1.0.2 Using GI Clone I used the following methodology to upgrade from 11.2.0.4 to 12.1.0.2 RAC.  This is particularly useful from a time perspective to avoid having to do a separate patching when it comes to applying bundle patches (Exadata) or PSUs.  I tested this process in my non-production environment and highly advise you to do the same to ensure the process is seamless. A) Prepare SOURCE GI home (In this case it is 12.1.0.2) to be cloned: ******************************** Follow documentation for "Preparing the Oracle Grid Infrastructure Home for Cloning" at https://docs.oracle.com/database/121/CWADD/clonecluster.htm#CWADD92116 Oracle® Clusterware Administration and Deployment Guide 12c Release 1 (12.1) Part Number E16794-16 ******************************** A01) Stop all databases via srvctl (run by oracle) srvctl stop database -d {db_unique_name} A02) Stop CRS (run by root) export ORACLE_HO...
Read more

RMAN-10038: database session for channel prm3 terminated unexpectedly

Problem: I had an issue where I was trying to build a 1.7 TB Data Guard environment and kept receiving the highlighted error below.  I thought it was some type of fluke so I restarted the job 2 more times only to receive the same error. Fix: After researching the issue, I found it was because the sqlnet.ora needed to be configured correctly.  I added the entry " SQLNET.EXPIRE_TIME = 10 " to the sqlnet.ora on both the source and the target database servers under the RDBMS $ORACLE_HOME/network/admin.  Upon restarting the RMAN active database standby build, it was able to move past this error and built successfully. Starting backup at 07-MAY-17 channel prm1: starting datafile copy input datafile file number=00003 name=+DATA/testdb/datafile/undotbs1.262.940412881 channel prm2: starting datafile copy input datafile file number=00129 name=+DATA/testdb/datafile/welldoc1.390.940509955 channel prm3: starting datafile copy input datafile file number=00130 name=+DATA/...
Read more