1. 2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN
Oracle RAC, Data Guard, and Pluggable Databases: When MAA Meets Multitenant
Ludovico CALDARA Oracle ACE, Senior Consultant Trivadis AG

2. 2013 © Trivadis
About Ludovico Caldara
28.09.2014
When MAA Meets Multitenant
2
■14 Years DBA (started with 7.3.4)
▪I do it everywhere (also on Windows)
■RAC ATTACK Ninja & co-writer
■RAC SIG Board member
■OCP (11g, 12c, MySQL) & OCE
■Italian living in Switzerland
■http://www.ludovicocaldara.net
■@ludodba
■ludovicocaldara

3. 2013 © Trivadis
Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany and Austria. We offer our services in the following strategic business fields: Trivadis Services takes over the interacting operation of your IT systems.
Our company
O P E R A T I O N
28.09.2014
When MAA Meets Multitenant
3

4. 2013 © Trivadis
With over 600 specialists and IT experts in your region
4
11 Trivadis branches and more than 600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget: CHF 5.0 / EUR 4 million
Financially self-supporting and sustainably profitable
Experience from more than 1,900 projects per year at over 800 customers
Hamburg
Düsseldorf
Frankfurt
Freiburg
Munich
Vienna
Basle
Zurich
Bern
Lausanne
4
Stuttgart
28.09.2014
When MAA Meets Multitenant
4

5. 2013 © Trivadis
AGENDA
1.RAC and Multitenant
Why and how
Dealing with PDBs and services
Demo
2.RAC, DG and MULTITENANT
Why and how
Dealing with PDBs and services
Demo
28.09.2014
When MAA Meets Multitenant
5

6. 2013 © Trivadis
28.09.2014
When MAA Meets Multitenant
Part I
6

7. 2013 © Trivadis
28.09.2014
When MAA Meets Multitenant
RAC and Multitenant
Why and How
7

8. 2013 © Trivadis
Multitenant on a single instance: is it worth?
28.09.2014
When MAA Meets Multitenant
8
Consolidate on one CDB, one server

9. 2013 © Trivadis
Multitenant on a single instance: is it worth?
28.09.2014
When MAA Meets Multitenant
9
Consolidate on one CDB, one server
The server fills up? Prepare another server and CDB!

10. 2013 © Trivadis
Multitenant on a single instance: is it worth?
28.09.2014
When MAA Meets Multitenant
10
Consolidate on one CDB, one server
The server fills up? Prepare another server and CDB!
Create new PDBs on the new server

11. 2013 © Trivadis
Multitenant on a single instance: is it worth?
28.09.2014
When MAA Meets Multitenant
11
Consolidate on one CDB, one server
The server fills up? Prepare another server and CDB!
Create new PDBs on the new server, REPEAT!

12. 2013 © Trivadis
Multitenant on a single instance: is it worth?
28.09.2014
When MAA Meets Multitenant
12
Consolidate on one CDB, one server
The server fills up? Prepare another server and CDB!
Create new PDBs on the new server, REPEAT!
A PDB increases? Need to make room for the new workload

13. 2013 © Trivadis
Multitenant on a single instance: is it worth?
28.09.2014
When MAA Meets Multitenant
13
Is that consolidation?

14. 2013 © Trivadis
Multitenant on a single instance: is it worth?
28.09.2014
When MAA Meets Multitenant
14
Is that consolidation? What if you have to change static parameters?

15. 2013 © Trivadis
Multitenant on a single instance: is it worth?
28.09.2014
When MAA Meets Multitenant
15
Is that consolidation? What if you have to change static parameters?
Bouncing your single-instance CDB takes offline all your PDBs

16. 2013 © Trivadis
Multitenant on a single instance: is it worth?
28.09.2014
When MAA Meets Multitenant
16
Is that consolidation? What if you have to change static parameters?
Bouncing your single-instance CDB takes offline all your PDBs
You would never stop all your services and reboot the instance, right?
Well, on a single-instance you have to do it!

17. 2013 © Trivadis
RAC empowers Multitenant
28.09.2014
When MAA Meets Multitenant
17
With RAC+MT, the Container Database itself is a RAC Database

18. 2013 © Trivadis
RAC empowers Multitenant
28.09.2014
When MAA Meets Multitenant
18
With RAC+MT, the Container Database itself is a RAC Database
The PDBs are still available in case of instance crash

19. 2013 © Trivadis
RAC empowers Multitenant
28.09.2014
When MAA Meets Multitenant
19
With RAC+MT, the Container Database itself is a RAC Database
The PDBs are still available in case of instance crash
Instances can be bounced one at a time without affecting the CDB

20. 2013 © Trivadis
RAC empowers Multitenant
28.09.2014
When MAA Meets Multitenant
20
The CDB can accommodate new PDBs by adding new servers/instances

21. 2013 © Trivadis
RAC empowers Multitenant
28.09.2014
When MAA Meets Multitenant
21
The CDB can accommodate new PDBs by adding new servers/instances
The workload is distributed across instances. How?

22. 2013 © Trivadis
28.09.2014
When MAA Meets Multitenant
RAC and Multitenant Dealing with PDBs and Services
22

23. 2013 © Trivadis
The services are the key!
28.09.2014
When MAA Meets Multitenant
23
The PDBs are mounted by default. Starting the service makes the cluster open the PDB!
Services on CDBRAC_1
Services on CDBRAC_2
Cache on instance CDBRAC_2
Cache on instance CDBRAC_1

24. 2013 © Trivadis
The services are the key!
28.09.2014
When MAA Meets Multitenant
24
Singleton services open the PDBs only on the nodes they run on.
Only the blocks of accessed PDBs will populate one instance’s cache.
Services on CDBRAC_1
Services on CDBRAC_2
Cache on instance CDBRAC_2
Cache on instance CDBRAC_1

25. 2013 © Trivadis
The services are the key!
28.09.2014
When MAA Meets Multitenant
25
Access big PDBs from more than one instance
Either declare the service UNIFORM or create a new one for the same PDB
Services on CDBRAC_1
Services on CDBRAC_2
Cache on instance CDBRAC_2
Cache on instance CDBRAC_1

26. 2013 © Trivadis
The services are the key!
28.09.2014
When MAA Meets Multitenant
26
Need more room/power?
Services on CDBRAC_1
Services on CDBRAC_2
Cache on instance CDBRAC_2
Cache on instance CDBRAC_1

27. 2013 © Trivadis
The services are the key!
28.09.2014
When MAA Meets Multitenant
27
Need more room/power?
Add an instance and modify/relocate some services
Services on CDBRAC_1
Services on CDBRAC_3
Cache on CDBRAC_3
Cache on CDBRAC_1
Services on CDBRAC_2
Cache on CDBRAC_2

28. 2013 © Trivadis
The services are the key!
28.09.2014
When MAA Meets Multitenant
28
Need more room/power?
Add an instance and modify/relocate some services
No need of unplugging/plugging PDBs! They’re in the same CDB!
Services on CDBRAC_1
Cache on CDBRAC_3
Cache on CDBRAC_1
Cache on CDBRAC_2
Services on CDBRAC_3
Services on CDBRAC_2

29. 2013 © Trivadis
The services are the key!
28.09.2014
When MAA Meets Multitenant
29
If you stop a service
Services on CDBRAC_1
Services on CDBRAC_3
Services on CDBRAC_2
Cache on CDBRAC_3
Cache on CDBRAC_1
Cache on CDBRAC_2

30. 2013 © Trivadis
The services are the key!
28.09.2014
When MAA Meets Multitenant
30
If you stop a service
The PDB is still open but not accessible
Cache on CDBRAC_3
Cache on CDBRAC_1
Cache on CDBRAC_2
Services on CDBRAC_1
Services on CDBRAC_3
Services on CDBRAC_2

31. 2013 © Trivadis
The services are the key!
28.09.2014
When MAA Meets Multitenant
31
If you stop a service
The PDB is still open but not accessible
Its blocks are aged out from the buffer cache using the LRU mechanism
Services on CDBRAC_1
Services on CDBRAC_3
Services on CDBRAC_2
Cache on CDBRAC_3
Cache on CDBRAC_1
Cache on CDBRAC_2

32. 2013 © Trivadis
Service Failover
28.09.2014
When MAA Meets Multitenant
32
After instance crash, a singleton service fails over
Services on CDBRAC_1
Services on CDBRAC_2
Cache on instance CDBRAC_2
Cache on instance CDBRAC_1

33. 2013 © Trivadis
Service Failover
28.09.2014
When MAA Meets Multitenant
33
After instance crash, a singleton service fails over
Services on CDBRAC_1
Services on CDBRAC_2
Cache on instance CDBRAC_2
Cache on instance CDBRAC_1

34. 2013 © Trivadis
Srvctl syntax for services and PDBs
srvctl add service –db db_unique_name –service service_name –serverpool server_pool –cardinality uniform –role primary –failovertype select –failovermethod basic -policy automatic -failoverdelay 2 –failoverretry 180 -pdb pluggable_database
srvctl modify service … -pdb pluggable_database
There’s an overall limit of 512 services per CDB, be careful!
28.09.2014
When MAA Meets Multitenant
34

35. 2013 © Trivadis
Many PDBs, many services, many changes
Important: Oracle recommends that you limit configuration changes to the minimum requirement and that you not perform other service operations while the online service modification is in progress. http://docs.oracle.com/database/121/RACAD/srvctladmin.htm#RACAD005
28.09.2014
When MAA Meets Multitenant
35

36. 2013 © Trivadis
RAC and Multitenant: Little demo?
28.09.2014
When MAA Meets Multitenant
36
CDBATL_2 raca01
CDBATL_1
raca02
CDBATL
MAAZ

37. 2013 © Trivadis
28.09.2014
When MAA Meets Multitenant
Part II
37

38. 2013 © Trivadis
28.09.2014
When MAA Meets Multitenant
RAC, DG and Multitenant
Why and How
38

39. 2013 © Trivadis
RAC, Data Guard & Multitenant, why?
28.09.2014
When MAA Meets Multitenant
39
The reasons of having a Data Guard architecture are the same…

40. 2013 © Trivadis
RAC, Data Guard & Multitenant, why?
28.09.2014
When MAA Meets Multitenant
40
The reasons of having a Data Guard architecture are the same…

41. 2013 © Trivadis
Multitenant comes with a great benefit…
28.09.2014
When MAA Meets Multitenant
41
Single configuration, simplified administration
CDBATL_1
CDBATL_2
CDBATL
MAAZ
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBGVA_1
CDBGVA_2
CDBGVA
MAAZ
MRP0

42. 2013 © Trivadis
Multitenant comes with a great benefit…
28.09.2014
When MAA Meets Multitenant
42
Single configuration, simplified administration
CDBATL_1
CDBATL_2
CDBATL
MAAZ
LUDO
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBGVA_1
CDBGVA_2
CDBGVA
MAAZ
MRP0

43. 2013 © Trivadis
Multitenant comes with a great benefit…
28.09.2014
When MAA Meets Multitenant
43
Single configuration, simplified administration
CDBATL_1
CDBATL_2
CDBATL
MAAZ
LUDO
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBGVA_1
CDBGVA_2
CDBGVA
MAAZ
LUDO
MRP0

44. 2013 © Trivadis
… or limitation?
28.09.2014
When MAA Meets Multitenant
44
Cannot have different PDBs with different roles
CDBATL_1
CDBATL_2
CDBATL
MAAZ
LUDO
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBGVA_1
CDBGVA_2
CDBGVA
MAAZ
LUDO
MRP0

45. 2013 © Trivadis
How?
28.09.2014
When MAA Meets Multitenant
45
ONE CDB = ONE Data Guard configuration

46. 2013 © Trivadis
How?
28.09.2014
When MAA Meets Multitenant
46
ONE CDB = ONE Data Guard configuration
All Data Guard commands (alter database or broker commands) are executed while connected to the CDB$ROOT
CDBATL_1
CDBATL_2
CDBATL (CDB$ROOT)
MAAZ
LUDO

47. 2013 © Trivadis
PDB Creation from PDB$SEED
28.09.2014
When MAA Meets Multitenant
47
SQL> create pluggable database MAAZ;
CDBATL_1
CDBATL_2
CDBATL
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBATL_1
CDBATL_2
CDBATL
MRP0
SEED
SEED

48. 2013 © Trivadis
PDB Creation from PDB$SEED
28.09.2014
When MAA Meets Multitenant
48
SQL> create pluggable database MAAZ;
CDBATL_1
CDBATL_2
CDBATL
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBATL_1
CDBATL_2
CDBATL
MRP0
SEED
SEED
MAAZ

49. 2013 © Trivadis
PDB Creation from PDB$SEED
28.09.2014
When MAA Meets Multitenant
49
SQL> create pluggable database MAAZ;
CDBATL_1
CDBATL_2
CDBATL
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBATL_1
CDBATL_2
CDBATL
MRP0
SEED
SEED
MAAZ
MAAZ
Recovery created pluggable database MAAZ
Recovery copied files for tablespace SYSTEM
Recovery successfully copied file +DATA/CDBGVA/…/DATAFILE/system.435.856973955 from +DATA/CDBGVA/…/DATAFILE/system.280.855055053
Successfully added datafile 24 to media recovery

50. 2013 © Trivadis
PDB Creation from PDB$SEED
28.09.2014
When MAA Meets Multitenant
50
SQL> create pluggable database MAAZ;
CDBATL_1
CDBATL_2
CDBATL
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBATL_1
CDBATL_2
CDBATL
MRP0
SEED
SEED
MAAZ
MAAZ
Recovery created pluggable database MAAZ Recovery copied files for tablespace SYSTEM Recovery successfully copied file +DATA/CDBGVA/…/DATAFILE/system.435.856973955 from +DATA/CDBGVA/…/DATAFILE/system.280.855055053 Successfully added datafile 24 to media recovery
MOUNTED OR OPEN READ ONLY

51. 2013 © Trivadis
PDB Creation from other PDB
28.09.2014
When MAA Meets Multitenant
51
SQL> create pluggable database LUDO FROM MAAZ;
CDBATL_1
CDBATL_2
CDBATL
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBATL_1
CDBATL_2
CDBATL
MRP0
MAAZ
MAAZ

52. 2013 © Trivadis
PDB Creation from other PDB
28.09.2014
When MAA Meets Multitenant
52
SQL> create pluggable database LUDO FROM MAAZ;
CDBATL_1
CDBATL_2
CDBATL
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBATL_1
CDBATL_2
CDBATL
MRP0
MAAZ
MAAZ
LUDO

53. 2013 © Trivadis
PDB Creation from other PDB
28.09.2014
When MAA Meets Multitenant
53
SQL> create pluggable database LUDO FROM MAAZ;
CDBATL_1
CDBATL_2
CDBATL
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBATL_1
CDBATL_2
CDBATL
MRP0
MAAZ
MAAZ
LUDO
LUDO
Recovery created pluggable database LUDO Recovery copied files for tablespace SYSTEM Recovery successfully copied file +DATA/CDBGVA/…/DATAFILE/system.362.85568186 from +DATA/CDBGVA/…/DATAFILE/system.435.856973955 Successfully added datafile 26 to media recovery

54. 2013 © Trivadis
PDB Creation from other PDB
28.09.2014
When MAA Meets Multitenant
54
SQL> create pluggable database LUDO FROM MAAZ;
CDBATL_1
CDBATL_2
CDBATL
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBATL_1
CDBATL_2
CDBATL
MRP0
MAAZ
MAAZ
LUDO
LUDO
Recovery created pluggable database LUDO Recovery copied files for tablespace SYSTEM Recovery successfully copied file +DATA/CDBGVA/…/DATAFILE/system.362.85568186 from +DATA/CDBGVA/…/DATAFILE/system.435.856973955 Successfully added datafile 26 to media recovery
ONLY IF OPEN READ ONLY

55. 2013 © Trivadis
PDB Creation from other PDB
28.09.2014
When MAA Meets Multitenant
55
SQL> create pluggable database LUDO FROM MAAZ;
CDBATL_1
CDBATL_2
CDBATL
LGWR
LGWR
LNS
LNS
RFS
RFS
CDBATL_1
CDBATL_2
CDBATL
MRP0
MAAZ
MAAZ
LUDO
LUDO
Recovery created pluggable database LUDO
Recovery copied files for tablespace SYSTEM
Recovery successfully copied file +DATA/CDBGVA/…/DATAFILE/system.362.85568186 from +DATA/CDBGVA/…/DATAFILE/system.435.856973955
Successfully added datafile 26 to media recovery
ONLY IF OPEN READ ONLY
ACTIVE DATAGUARD!!!

56. 2013 © Trivadis
PDB Creation from other PDB WITHOUT ADG
28.09.2014
When MAA Meets Multitenant
56
Documentation says: copy the datafiles on the standby before the clone

57. 2013 © Trivadis
PDB Creation from other PDB WITHOUT ADG
28.09.2014
When MAA Meets Multitenant
57
Documentation says: copy the datafiles on the standby before the clone
But RAC => ASM => OMF => Cannot guess the file names!!
Need to workout a solution so that you don’t mess your OMF with aliases

58. 2013 © Trivadis
PDB Creation from other PDB WITHOUT ADG
28.09.2014
When MAA Meets Multitenant
58
Documentation says: copy the datafiles on the standby before the clone
But RAC => ASM => OMF => Cannot guess the file names!!
Need to workout a solution so that you don’t mess your OMF with aliases
There are different solutions, too complex to cover here. I’ll just show the easiest one.

59. 2013 © Trivadis
PDB Creation from other PDB WITHOUT ADG *
28.09.2014
When MAA Meets Multitenant
59
SQL> create pluggable database LUDO from MAAZ standbys=NONE;
Backup/Restore PDB from Primary to Standby over the network
Stop Recovery
Alter pluggable database LUDO enable recovery;
Start Recovery
* Not the official way, but much easier IMHO with ASM

60. 2013 © Trivadis
Restore PDB from Primary to Standby
28.09.2014
When MAA Meets Multitenant
60
PRIMARY RMAN> backup as copy pluggable database LUDO format '/tmp/ludo%f.dbf'; $ scp /tmp/ludo*.dbf racb01:/tmp
STANDBY RMAN> catalog start with ‘/tmp/ludo’ RMAN> set newname for pluggable database LUDO to new; RMAN> restore pluggable database LUDO; RMAN> switch pluggable database LUDO to copy;

61. 2013 © Trivadis
28.09.2014
When MAA Meets Multitenant
RAC, DG and Multitenant Dealing with PDBs and services
61

62. 2013 © Trivadis
Srvctl syntax for services and PDBs with ADG
srvctl add service –db db_unique_name –service ro_service_name –serverpool server_pool –cardinality uniform –role pyhsical_standby –failovertype select –failovermethod basic -policy automatic -failoverdelay 2 –failoverretry 180 -pdb pluggable_database
Create the same service on the Primary first, so it exists and can run on the standby
Many services per PDB: don’t forget the 512 service limit per CDB!!
28.09.2014
When MAA Meets Multitenant
62

63. 2013 © Trivadis
Connection descriptor for MAA
LUDOAPP = (DESCRIPTION_LIST= (LOAD_BALANCE=off) (FAILOVER=on) (DESCRIPTION = (CONNECT_TIMEOUT=5) (TRANSPORT_CONNECT_TIMEOUT=3) (RETRY_COUNT=3) (ADDRESS_LIST= (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST = raca-scan)(PORT = 1521))) (CONNECT_DATA = (SERVICE_NAME = LUDOAPP)) ) (DESCRIPTION = (CONNECT_TIMEOUT=5) (TRANSPORT_CONNECT_TIMEOUT=3) (RETRY_COUNT=3) (ADDRESS_LIST= (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST = racb-scan)(PORT = 1521))) (CONNECT_DATA = (SERVICE_NAME = LUDOAPP)) ) )
28.09.2014
When MAA Meets Multitenant
63

64. 2013 © Trivadis
RAC, DG and Multitenant: Little demo?
28.09.2014
When MAA Meets Multitenant
64
raca01
raca02
racb01
racb02
CDBATL
Data Guard
CDBGVA

65. 2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN
?????????

66. 2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN
Thank you!
Ludovico Caldara Oracle ACE, Senior Consultant Trivadis AG ludovico.caldara@trivadis.com @ludodba