一.前言 

        在我們生產(chǎn)環(huán)境中，dataguard通常用來(lái)提供容災(zāi)方案，因此有效地保護(hù)歸檔日志就顯得尤為重要，否則一旦形成日志gap，且歸檔日志被誤刪除，進(jìn)而導(dǎo)致DG會(huì)失去數(shù)據(jù)同步的能力，從這個(gè)角度來(lái)說(shuō)DG似乎又顯得十分脆弱。本文會(huì)簡(jiǎn)要地提到如何制定較好的備份策略，但是最重要的目的是來(lái)看看如何修復(fù)gap?；蛟S有人想到了重建DG，特別是在11g可以在線(xiàn)重建，但是在真正的生產(chǎn)環(huán)境，重建的成本是很高的，是因?yàn)椋?/p>

1. 生產(chǎn)環(huán)境通常數(shù)據(jù)庫(kù)體積龐大，重建過(guò)程中需要傳輸全備文件到standby，即使使用了壓縮，其體積可能仍然非常龐大。

2.大部分生產(chǎn)系統(tǒng)的DG是跨地域的，這意味著primary與standby端的帶寬可能及其有限，通常正常傳輸速度達(dá)到1M/s已經(jīng)算很不錯(cuò)。但是對(duì)于傳輸龐大的備份集，仍然是杯水車(chē)薪。還不算在傳輸期間primary產(chǎn)生的歸檔日志也是需要后續(xù)傳輸?shù)?。其時(shí)間可能遠(yuǎn)遠(yuǎn)長(zhǎng)于僅傳輸全備的時(shí)間。

3.再說(shuō)說(shuō)11g的duplicate from active database,這個(gè)方法固然簡(jiǎn)單，但是此法在傳輸時(shí)是不能壓縮的！在帶寬有限，數(shù)據(jù)庫(kù)體積龐大的情況下，基本上是不能在限定的時(shí)間內(nèi)完成創(chuàng)建的。這個(gè)方法所需的時(shí)間預(yù)計(jì)會(huì)比手動(dòng)壓縮傳輸?shù)姆椒ㄩL(zhǎng)6倍以上！也是不可取的。

因此正確的修復(fù)gap的方式，是利用增量備份集，在standby端進(jìn)行恢復(fù)。

二.備份策略因?yàn)樵隽總浞莸拇嬖诙屿`活、實(shí)用

     最佳的備份策略能在備份所需空間與恢復(fù)時(shí)間上達(dá)到平衡，具體根據(jù)實(shí)際業(yè)務(wù)情況而定，但是本質(zhì)上就是利用全備+增量備份+日志備份的方式。

     關(guān)于增量備份的官方詳細(xì)內(nèi)容，請(qǐng)參看另一篇博文 “Incremental Backups in Oracle 10.2.0.5” Translation & Notes 

     其中一個(gè)重要的概念就是：

    A level 1 incremental backup can be either of the following types:
    A differential backup, which backs up all blocks changed after the most recent incremental backup at level 1 or 0
    A cumulative backup, which backs up all blocks changed after the most recent incremental backup at level 0   

    但是我們必須知道，所謂增量，實(shí)際上是基于SCN判斷的！

    因此舉例來(lái)說(shuō)：differential backup適用于dailybackup，而cumulative backup適用于weeklybackup，而零級(jí)備份適用于monthly backup或者更長(zhǎng)周期的全備。

    因此我目前的情況，OLTP系統(tǒng)，日志產(chǎn)生比較頻繁，比較傾向的一種的備份方式是：

    一周以?xún)?nèi)：

每月的第一天做上個(gè)月的零級(jí)全備。

這樣在任意時(shí)間需要恢復(fù)數(shù)據(jù)是，最多用到一個(gè)全備+一個(gè)周備+7個(gè)日備+一天之內(nèi)的歸檔日志。

特別注意的是：如果生產(chǎn)庫(kù)非常大，那么強(qiáng)烈建議使用block_change_tracking特性，能極大縮短增量備份時(shí)間，否則做一次增量備份的時(shí)間與做一次全備所需時(shí)間無(wú)異！

三.利用增量備份修復(fù)DG實(shí)例演示

Step1.場(chǎng)景模擬

OS: Linux5.8 x64    DB:Oracle 11.2.0.3 x64

主庫(kù)：test3_A  備庫(kù)test3_B   且正常運(yùn)行中

test3_B:

SQL> archive log list
Database log mode           Archive Mode
Automatic archival           Enabled
Archive destination           /u01/archivelogs/test3
Oldest online log sequence     45
Next log sequence to archive   0
Current log sequence           59

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>

test3_A:

SQL> archive log list
Database log mode           Archive Mode
Automatic archival           Enabled
Archive destination           /u01/archivelog/test3
Oldest online log sequence     57
Next log sequence to archive   59
Current log sequence           59
SQL> alter system switch logfile;

System altered.

SQL> /

System altered.

SQL> /

System altered.

SQL> archive log list
Database log mode           Archive Mode
Automatic archival           Enabled
Archive destination           /u01/archivelog/test3
Oldest online log sequence     60
Next log sequence to archive   62
Current log sequence           62

SQL> quit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@redhat ~]$ rman target /

Recovery Manager: Release 11.2.0.3.0 - Production on Mon Jun 10 21:31:04 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: TEST3 (DBID=915537947)

RMAN> list archivelog low sequence 60 high sequence 62;

using target database control file instead of recovery catalog
List of Archived Log Copies for database with db_unique_name TEST3_A
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - ---------
468     1    60      A 10-JUN-13
        Name: /u01/archivelog/test3/1_60_800115694.dbf

469     1    61      A 10-JUN-13
        Name: /u01/archivelog/test3/1_61_800115694.dbf

RMAN> delete archivelog low sequence 60 high sequence 61;      (相比10g DG的改進(jìn)是，沒(méi)有被applied的歸檔，是不能僅僅被delete命令刪除的，必須使用delete force刪除)

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=55 device type=DISK
RMAN-08120: WARNING: archived log not deleted, not yet applied by standby
archived log file name=/u01/archivelog/test3/1_60_800115694.dbf thread=1 sequence=60
RMAN-08120: WARNING: archived log not deleted, not yet applied by standby
archived log file name=/u01/archivelog/test3/1_61_800115694.dbf thread=1 sequence=61

RMAN> delete force archivelog low sequence 60 high sequence 61;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=55 device type=DISK
List of Archived Log Copies for database with db_unique_name TEST3_A
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - ---------
468     1    60      A 10-JUN-13
        Name: /u01/archivelog/test3/1_60_800115694.dbf

469     1    61      A 10-JUN-13
        Name: /u01/archivelog/test3/1_61_800115694.dbf


Do you really want to delete the above objects (enter YES or NO)? yes
deleted archived log
archived log file name=/u01/archivelog/test3/1_60_800115694.dbf RECID=468 STAMP=817766882
deleted archived log
archived log file name=/u01/archivelog/test3/1_61_800115694.dbf RECID=469 STAMP=817766885
Deleted 2 objects

RMAN> quit

Recovery Manager complete.
[oracle@redhat ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Jun 10 21:48:31 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter system switch logfile;

System altered.

SQL> /

System altered.

test3_B:

SQL> startup
ORACLE instance started.

Total System Global Area  622149632 bytes
Fixed Size            2230912 bytes
Variable Size          180356480 bytes
Database Buffers      436207616 bytes
Redo Buffers            3354624 bytes
Database mounted.
Database opened.
SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered.

此時(shí)警告日志文件中出現(xiàn)：

Mon Jun 10 21:48:27 2013
FAL[client]: Failed to request gap sequence
 GAP - thread 1 sequence 60-61
 DBID 915537947 branch 800115694
FAL[client]: All defined FAL servers have been attempted.
------------------------------------------------------------
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that's sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
------------------------------------------------------------
Mon Jun 10 21:48:40 2013
RFS[2]: Selected log 5 for thread 1 sequence 64 dbid 915537947 branch 800115694
Mon Jun 10 21:48:40 2013
Archived Log entry 76 added for thread 1 sequence 63 ID 0x37014efb dest 1:
RFS[2]: Selected log 4 for thread 1 sequence 65 dbid 915537947 branch 800115694
Mon Jun 10 21:48:44 2013
Archived Log entry 77 added for thread 1 sequence 64 ID 0x37014efb dest 1:

然后查看一下日志gap的范圍，發(fā)現(xiàn)正好是在primary強(qiáng)行刪掉的兩個(gè)歸檔日志：

SQL> select * from v$archive_gap;

   THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#
---------- ------------- --------------
     1          60         61

Step2.查看GAP的起始SCN信息

       歸檔日志的gap，具體來(lái)說(shuō)就是從某個(gè)scn開(kāi)始，備庫(kù)沒(méi)有可用的歸檔日志進(jìn)行apply。但是盡管如此，后來(lái)的歸檔日志仍然會(huì)自動(dòng)傳輸?shù)絪tandby端，但是卻不能被應(yīng)用，也就不能被rman的備份策略自動(dòng)刪除（force除外，但是我相信生產(chǎn)系統(tǒng)幾乎不會(huì)用到它），進(jìn)而導(dǎo)致空間占滿(mǎn)問(wèn)題；相應(yīng)的primary端的日志同樣不能被正常刪除，也會(huì)遇到空間問(wèn)題，然后可能hang住。這對(duì)于磁盤(pán)空間本不富裕，且日志產(chǎn)生頻繁的系統(tǒng)就是一顆定時(shí)炸彈！

那么先來(lái)查看一下standby （test3_B）端截至目前的scn：

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
    3221814

那么意味著，我們必須至少將standby的scn繼續(xù)向后推動(dòng)到primary在gap之后的最早的一個(gè)日志（可能是在線(xiàn)日志，也可能是歸檔日志）的first_change#，才能繼續(xù)應(yīng)用日志！ 因此接下來(lái)需要在primary端做一次 基于SCN起始的增量備份

Step3.定制增量備份集，并傳輸?shù)絪tandby端

在test3_A做基于SCN的增量備份，該備份包含了從3221814起，截至數(shù)據(jù)文件當(dāng)前時(shí)間點(diǎn)，所有變化的數(shù)據(jù)塊。如果此時(shí)primary沒(méi)有全備也沒(méi)關(guān)系。但是如果有block_change_tracking的話(huà)，能顯著縮短這一過(guò)程的時(shí)間！

注意下面會(huì)同時(shí)生成一份最新的standby controlfile的備份集。這是一個(gè)自定義跨度的增量備份，因此differential 或者cumulative或者level都不需要指定了！

test3_A:

RMAN>run

}

注：或者加上tag能夠方便定位具體備份集

[oracle@redhat ~]$ cd /u01/rman/test3/
[oracle@redhat test3]$ ls -l
total 34564
-rw-r----- 1 oracle dba 11600384 Jun 10 22:17 block_tracking.log
-rw-r----- 1 oracle dba 10420224 Nov 23  2012 c-915537947-20121123-00
-rw-r----- 1 oracle dba 11051008 Jun 10 22:16 db_25obsb96_20130610.bk
-rw-r----- 1 oracle dba  1130496 Jun 10 22:16 db_26obsb9l_20130610.bk
-rw-r----- 1 oracle dba  1130496 Jun 10 22:16 db_27obsb9p_20130610.bk
[oracle@redhat test3]$ scp *20130610.bk oracle@10.0.0.21:/u01/rman/test3/
oracle@10.0.0.21's password:
db_25obsb96_20130610.bk                                                                                                                                        100%   11MB  10.5MB/s   00:01    
db_26obsb9l_20130610.bk                                                                                                                                        100% 1104KB   1.1MB/s   00:00    
db_27obsb9p_20130610.bk                                                                                                                                        100% 1104KB   1.1MB/s   00:00    
[oracle@redhat test3]$

最后確定主庫(kù)在此scn后是否添加過(guò)數(shù)據(jù)文件，如果有則需要在standby端也手動(dòng)添加：

SQL> select FILE#,name from v$datafile where CREATION_CHANGE#> =3221814;

no rows selected

Step4.停止備庫(kù)日志應(yīng)用，在stanby端使用增量備份recover數(shù)據(jù)文件，并恢復(fù)standby controlfile以越過(guò)gap

SQL> alter database recover managed standby database cancel;

Database altered.

RMAN> shutdown immediate

using target database control file instead of recovery catalog
database closed
database dismounted
Oracle instance shut down

RMAN> startup mount

connected to target database (not started)
Oracle instance started
database mounted

Total System Global Area     622149632 bytes

Fixed Size                     2230912 bytes
Variable Size                180356480 bytes
Database Buffers             436207616 bytes
Redo Buffers                   3354624 bytes

RMAN> catalog start with '/u01/rman/test3';

searching for all files that match the pattern /u01/rman/test3

List of Files Unknown to the Database
=====================================
File Name: /u01/rman/test3/db_25obsb96_20130610.bk
File Name: /u01/rman/test3/db_27obsb9p_20130610.bk
File Name: /u01/rman/test3/db_36nm49ec_20120925.bk
File Name: /u01/rman/test3/c-915537947-20120925-00
File Name: /u01/rman/test3/c-915537947-20121030-01
File Name: /u01/rman/test3/c-915537947-20121106-00
File Name: /u01/rman/test3/db_37nm49go_20120925.bk
File Name: /u01/rman/test3/c-915537947-20121030-02
File Name: /u01/rman/test3/db_26obsb9l_20130610.bk
File Name: /u01/rman/test3/arc_35nm49e8_20120925.bk
File Name: /u01/rman/test3/arc_38nm49gr_20120925.bk

Do you really want to catalog the above files (enter YES or NO)? yes
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /u01/rman/test3/db_25obsb96_20130610.bk
File Name: /u01/rman/test3/db_27obsb9p_20130610.bk
File Name: /u01/rman/test3/db_36nm49ec_20120925.bk
File Name: /u01/rman/test3/c-915537947-20120925-00
File Name: /u01/rman/test3/c-915537947-20121030-01
File Name: /u01/rman/test3/c-915537947-20121106-00
File Name: /u01/rman/test3/db_37nm49go_20120925.bk
File Name: /u01/rman/test3/c-915537947-20121030-02
File Name: /u01/rman/test3/db_26obsb9l_20130610.bk
File Name: /u01/rman/test3/arc_35nm49e8_20120925.bk
File Name: /u01/rman/test3/arc_38nm49gr_20120925.bk

RMAN> recover database noredo;

Starting recover at 10-JUN-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=26 device type=DISK
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: /u01/oradata/test3/system01.dbf
destination for restore of datafile 00002: /u01/oradata/test3/undotbs01.dbf
destination for restore of datafile 00003: /u01/oradata/test3/sysaux01.dbf
destination for restore of datafile 00004: /u01/oradata/test3/users01.dbf
destination for restore of datafile 00005: /u01/oradata/test3/example01.dbf
channel ORA_DISK_1: reading from backup piece /u01/rman/test3/db_25obsb96_20130610.bk
channel ORA_DISK_1: piece handle=/u01/rman/test3/db_25obsb96_20130610.bk tag=TAG20130610T221605
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:23

Finished recover at 10-JUN-13

RMAN> quit


Recovery Manager complete.

test3_B:
[oracle@redhat1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Jun 10 22:38:51 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
    3221814

test3_A:

SQL>  select thread#,sequence#,first_change#,next_change# from v$archived_log where first_change#<3221814 and next_change#>3221814；

THREAD#  SEQUENCE# FIRST_CHANGE# NEXT_CHANGE#
---------- ---------- ------------- ------------
     1       59        3221114     3221815
     1       59        3221114     3221815

test3_B:

RMAN> restore standby controlfile from '/u01/rman/test3/db_27obsb9p_20130610.bk';   （一定要恢復(fù)standby控制文件，否則standby DB不能跳過(guò)原有g(shù)ap的信息）

Starting restore at 10-JUN-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=19 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
output file name=/u02/test3/control01.ctl
output file name=/u03/test3/control02.ctl
Finished restore at 10-JUN-13

RMAN> quit


Recovery Manager complete.
[oracle@redhat1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Jun 10 23:13:10 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter database mount;

Database altered.

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-10458: standby database requires recovery
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '/u01/oradata/test3/system01.dbf'

SQL> recover standby database using backup controlfile;
ORA-00279: change 3230822 generated at 06/10/2013 22:16:06 needed for thread 1
ORA-00289: suggestion : /u01/archivelogs/test3/1_66_800115694.dbf
ORA-00280: change 3230822 for thread 1 is in sequence #66


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
auto
ORA-00279: change 3232949 generated at 06/10/2013 22:41:24 needed for thread 1
ORA-00289: suggestion : /u01/archivelogs/test3/1_67_800115694.dbf
ORA-00280: change 3232949 for thread 1 is in sequence #67
ORA-00278: log file '/u01/archivelogs/test3/1_66_800115694.dbf' no longer
needed for this recovery


ORA-00308: cannot open archived log '/u01/archivelogs/test3/1_67_800115694.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

SQL> alter database open;

Database altered.

SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered.

此時(shí)standby的警告日志信息，發(fā)現(xiàn)DG已經(jīng)可以越過(guò)gap，開(kāi)始從67號(hào)日志開(kāi)始apply：

Mon Jun 10 23:17:22 2013
Primary database is in MAXIMUM PERFORMANCE mode
Changing standby controlfile to MAXIMUM PERFORMANCE mode
RFS[1]: Assigned to RFS process 26846
RFS[1]: Selected log 4 for thread 1 sequence 68 dbid 915537947 branch 800115694
Mon Jun 10 23:17:25 2013
RFS[2]: Assigned to RFS process 26850
RFS[2]: Selected log 5 for thread 1 sequence 67 dbid 915537947 branch 800115694
Mon Jun 10 23:17:28 2013
Archived Log entry 1 added for thread 1 sequence 67 ID 0x37014efb dest 1:
Mon Jun 10 23:18:59 2013
alter database recover managed standby database using current logfile disconnect from session
Attempt to start background Managed Standby Recovery process (test3)
Mon Jun 10 23:18:59 2013
MRP0 started with pid=27, OS id=26857
MRP0: Background Managed Standby Recovery process started (test3)
Serial Media Recovery started
Managed Standby Recovery starting Real Time Apply
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /u01/archivelogs/test3/1_67_800115694.dbf
Completed: alter database recover managed standby database using current logfile disconnect from session
Media Recovery Waiting for thread 1 sequence 68 (in transit)
Recovery of Online Redo Log: Thread 1 Group 4 Seq 68 Reading mem 0
  Mem# 0: /u02/test3/standby_redo04a.log
  Mem# 1: /u03/test3/standby_redo04b.log

至此恢復(fù)成功！

四. 值得注意的幾點(diǎn)

1. 如果使用手動(dòng)壓縮全備，并使用dulipcate target database for standy database的重建方式，即使有增量備份傳輸?shù)絪tandby，也不會(huì)被自動(dòng)應(yīng)用。

2. 如果先恢復(fù)standby controlfile, 就不用做catalog start with，接下來(lái)就能做recover database了。

3. 這是一個(gè)真實(shí)的案例！這個(gè)問(wèn)題在生產(chǎn)中遇到時(shí)，才會(huì)發(fā)現(xiàn)制約因素很多，遠(yuǎn)沒(méi)有本機(jī)測(cè)試環(huán)境那么順利。經(jīng)驗(yàn)的積累很重要！