Effective and Efficient Python with Oracle Database1. Effective and Efficient Python
with Oracle Database
Christopher Jones
Data Access Development
Oracle Database
Copyright © 2019 Oracle and/or its affiliates.
christopher.jones@oracle.com
@ghrd
blogs..oracle.com/opal
2. The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor
Copyright © 2019 Oracle and/or its affiliates.
3. Oracle Database for the Developer
Copyright © 2019 Oracle and/or its affiliates.
Third Party Open Source Drivers
Oracle Open Source Drivers
Oracle Call Interface (OCI)
Typically from Oracle Instant Client ZIPs or RPMs
ODPI-C
Also used by custom apps
Python
cx_Oracle
Node.js
node-oracledb
Go
goracle
Julia
Oracle.jl
Rust
rust-
oracle PHP
OCI8
PHP
PDO_OCI
R
ROracle
Go
go-oci8
Erlang
oranif Perl
DBD::
Oracle
Ruby
ruby-
oci8
Oracle Call Interface, Oracle C++ Call Interface, ODBC, JDBC, ODP.NET,
Pro*C, Pro*COBOL, SQLJ, OLE DB, OLE DB for OLAP
Oracle Proprietary Drivers
4. Themes
• Introduction & Installation
• cx_Oracle Feature Dive
• High Availability
• Towards Data Science
Copyright © 2019 Oracle and/or its affiliates.
6. Python in Four Points
• General purpose scripting language
• Huge package library with sophisticated packages
• “Everyone” is using it
• Great Oracle Database (and TimesTen) support
Copyright © 2019 Oracle and/or its affiliates.
7. cx_Oracle
• Open Source package for Python access to Oracle Database
• Covers all of Python Database API specification
• Many additions to support advanced Oracle features
• 50+ releases covering Oracle 8i through 19c
• Begun by Anthony Tuininga in 1998 for Oracle 8i and Python 1.5
• Latest release is cx_Oracle 7.2.3 (September 2019)
Copyright © 2019 Oracle and/or its affiliates.
8. cx_Oracle Features
Open Source
Maintained by Oracle
Hosted on PyPI and GitHub
Binary 'wheel' packages
SQL and PL/SQL Execution
Scrollable cursors
Extensive data type support
Binding to SQL objects, PL/SQL
index-by tables, and records
Fetching of large result sets
REF CURSORs
Copyright © 2019 Oracle and/or its affiliates.
Oracle Database JSON datatype
Large Objects: CLOBs and BLOBs
Simple Oracle Document Access
(SODA)
Transaction Management
Session Pooling
Database Resident Connection
Pooling (DRCP)
Privileged Connections and
Database startup/shutdown
External Authentication
Advanced Queuing (AQ)
Continuous Query Notification
(CQN)
Client Result Caching
End-to-end authentication and
tracing
Implicit Results
Transaction Guard
Edition Based Redefinition
Row Prefetching
Statement Caching
Application Context
Sharded Databases
Oracle Database High Availability
Features
Oracle Net Features including
Encryption
10. Installation in One Slide
1. Install Python 2.7 or 3.5+
2. Install cx_Oracle:
python -m pip install cx_Oracle --upgrade
--user
--proxy=http://proxy.example.com:80
3. Add Oracle 11.2 – 19c client libraries to the library search path
• Windows: PATH
• Linux: LD_LIBRARY_PATH or use ldconfig
• macOS: put files in ~/lib or /usr/local/lib
• Use Instant Client if your database is on a remote computer or the cloud
Copyright © 2019 Oracle and/or its affiliates.
11. Ok, Two Slides
4. For Cloud DBs, download wallet.zip
5. Extract and put files in instantclient_XX_Y/network/admin
cwallet.sso
sqlnet.ora
tnsnames.ora
6. Still need username/password at connection with cloud wallets
Copyright © 2019 Oracle and/or its affiliates.
13. Standalone Connections
import cx_Oracle
with cx_Oracle.connect("user", "password", "localhost/orclpdb1",
encoding="UTF-8") as connection:
cursor = connection.cursor()
for row in cursor.execute("select * from SomeTab"):
print(row)
Copyright © 2019 Oracle and/or its affiliates.
Python DB
14. Pooled Connections
pool = cx_Oracle.SessionPool("user", "password", "localhost/orclpdb1",
min=20, max=20, increment=0,
threaded=True,
encoding="UTF-8")
. . .
with pool.acquire() as connection:
cursor = connection.cursor()
for row in cursor.execute("select * from SomeTab"):
print(row)
Copyright © 2019 Oracle and/or its affiliates. Python DB
15. Many Apps Set Session State
with pool.acquire() as connection:
cursor = connection.cursor()
cursor.callproc("set_app_state")
do_work(connection)
Copyright © 2019 Oracle and/or its affiliates.
Round-trip required each
time connection is
acquired from the pool
Set NLS settings, time
zone, PL/SQL package
state, etc.
"A server round-trip is defined as the trip from the client to the
database server and back to the client."
16. Session Callback: All Connections with
Same State
def init_session(connection, requestedTag):
cursor = connection.cursor()
cursor.callproc("set_app_state")
pool = cx_Oracle.SessionPool("user", "password", "dsn", min=5,
max=20, increment=3, sessionCallback=init_session)
with pool.acquire() as connection:
do_work(connection)
Copyright © 2019 Oracle and/or its affiliates.
Only invoked for new
sessions or when the
requested tag does not
match the connection tag
Connection returned from
pool.acquire() already has
the right state
17. Getting Data Out of
the Database
Copyright © 2019 Oracle and/or its affiliates.
18. Fetching Rows (in Batches)
cursor = connection.cursor()
cursor.arraysize = 500
cursor.execute("select * from SomeTab")
while True:
rows = cursor.fetchmany()
if rows:
do_something(rows)
if len(rows) < cursor.arraysize:
break
Copyright © 2019 Oracle and/or its affiliates.
Default fetchmany() value.
Internal tuning buffer size for
fetchone() and fetchall() too.
19. Fetching LOBs
cursor = connection.cursor()
cursor.execute("select ClobVal from SomeTab")
for lob in cursor:
lobData = lob.read() # or str(lob)
Copyright © 2019 Oracle and/or its affiliates.
20. Fetching LOBs as Strings/Bytes
def handler(cursor, name, defaultType, size, precision, scale):
if defaultType == cx_Oracle.CLOB:
return cursor.var(cx_Oracle.LONG_STRING, arraysize=cursor.arraysize)
elif defaultType == cx_Oracle.BLOB:
return cursor.var(cx_Oracle.LONG_BINARY, arraysize=cursor.arraysize)
cursor.outputtypehandler = handler
cursor.execute("select LobVal from SomeTab")
for lobData in cursor:
...
Copyright © 2019 Oracle and/or its affiliates.
21. Fetching Numbers as Decimals
def handler(cursor, name, defaultType, size, precision, scale):
if defaultType == cx_Oracle.NUMBER:
return cursor.var(decimal.Decimal, arraysize=cursor.arraysize)
cursor.outputtypehandler = handler
for value, in cur.execute("select Data from DecimalTab"):
print("Value:", value, "* 3 =", value * 3)
Copyright © 2019 Oracle and/or its affiliates.
Default output: Output type handler output:
Value: 0.1 * 3 = 0.30000000000000004
Value: 3.1 * 3 = 9.3
Value: 7.1000000000000005 * 3 = 21.3
Value: 0.1 * 3 = 0.3
Value: 3.1 * 3 = 9.3
Value: 7.1 * 3 = 21.3
23. Binding Data
cursor = connection.cursor()
sql = "insert into MyTab (Id, Data) values (:idVal, :dataVal)"
# bind by position using a sequence (list or tuple)
cursor.execute(sql, [1, "String 1"])
cursor.execute(sql, (1, "String 1"))
# bind by name using a dictionary
cursor.execute(sql, {"idVal": 1, "dataVal": "String 1"})
# bind by name using keyword arguments
cursor.execute(sql, idVal=1, dataVal="String 1")
Copyright © 2019 Oracle and/or its affiliates.
24. Batch Execution
Copyright © 2019 Oracle and/or its affiliates.
Execute one DML or PL/SQL
statementwith many data values
rows = [ (1, "First" ),
(2, "Second" ),
(3, "Third" ),
(4, "Fourth" ),
(5, "Fifth" ),
(6, "Sixth" ),
(7, "Seventh" ) ]
cursor = connection.cursor()
cursor.setinputsizes(none, 20)
cursor.executemany("insert into MyTab (Id, Data) values (:idVal, :dataVal)", rows)
Saves “round-trips” between Python and Oracle Database
25. Batch Errors
dataToInsert = [
(1016, 10, 'Red'),
(1018, 20, 'Blue'),
(1018, 30, 'Green’), # duplicate key
(1022, 40, 'Yellow'),
(1021, 75, 'Orange'), # parent does not exist
]
cursor.executemany("insert into ChildTable values (:1, :2, :3)", dataToInsert,
batcherrors=True)
for error in cursor.getbatcherrors():
print("Error", error.message, "at row offset", error.offset)
Copyright © 2019 Oracle and/or its affiliates.
Error ORA-00001: unique constraint (PYTHONDEMO.CHILDTABLE_PK) violated at row offset 2
Error ORA-02291: integrity constraint (PYTHONDEMO.CHILDTABLE_FK) violated - parent key not
found at row offset 4
27. Binding Named Object Types
create table TestGeometry (
IntCol number(9) not null,
Geometry SDO_GEOMETRY not null
);
describe SDO_GEOMETRY
Name Null? Type
----------------------------------------- -------- ----------------------------
SDO_GTYPE NUMBER
. . .
SDO_ELEM_INFO MDSYS.SDO_ELEM_INFO_ARRAY
SDO_ORDINATES MDSYS.SDO_ORDINATE_ARRAY
. . .
Copyright © 2019 Oracle and/or its affiliates.
28. Binding Named Object Types (continued)
typeObj = conn.gettype("SDO_GEOMETRY")
elementInfoTypeObj = conn.gettype("SDO_ELEM_INFO_ARRAY")
ordinateTypeObj = conn.gettype("SDO_ORDINATE_ARRAY")
obj = typeObj.newobject() # or typeObj()
obj.SDO_GTYPE = 2003
obj.SDO_ELEM_INFO = elementInfoTypeObj.newobject()
obj.SDO_ELEM_INFO.extend([1, 1003, 3])
obj.SDO_ORDINATES = ordinateTypeObj.newobject()
obj.SDO_ORDINATES.extend([1, 1, 5, 7])
cur = conn.cursor()
cur.execute("insert into TestGeometry values (1, :objbv)", objbv=obj)
Copyright © 2019 Oracle and/or its affiliates.
29. Fetching Objects
def dumpobject(obj, prefix = " "):
if obj.type.iscollection:
print(prefix, "[")
for value in obj.aslist():
if isinstance(value, cx_Oracle.Object):
dumpobject(value, prefix + " ")
else:
print(prefix + " ", repr(value))
print(prefix, "]")
else:
print(prefix, "{")
for attr in obj.type.attributes:
value = getattr(obj, attr.name)
if isinstance(value, cx_Oracle.Object):
print(prefix + " " + attr.name + " :")
dumpobject(value, prefix + " ")
else:
print(prefix + " " + attr.name + " :", repr(value))
print(prefix, "}")
Copyright © 2019 Oracle and/or its affiliates.
31. Enqueuing
SQL> call dbms_aqadm.create_queue_table('RAW_QUEUE_TAB', 'RAW’);
SQL> call dbms_aqadm.create_queue('RAW_QUEUE', 'RAW_QUEUE_TAB’);
SQL> call dbms_aqadm.start_queue('RAW_QUEUE');
data = [ "Message 1", "Message 2", ... ]
queue = connection.queue("RAW_QUEUE")
# enqueue one at a time
for payloadData in data:
queue.enqOne(connection.msgproperties(payload=payloadData))
# enqueue many at a time
messages = [connection.msgproperties(payload=d) for d in data]
queue.enqMany(messages)
Copyright © 2019 Oracle and/or its affiliates.
32. Dequeuing
queue = connection.queue("RAW_QUEUE")
# dequeue one at a time
message = queue.deqOne()
print(message.payload.decode(connection.encoding))
# dequeue many at a time
maxNumMessages = 8
for message in queue.deqMany(maxNumMessages):
print(message.payload.decode(connection.encoding))
Copyright © 2019 Oracle and/or its affiliates.
34. Simple Oracle Document Access (SODA)
• NoSQL-style APIs to create and access documents
• Documents are often JSON
• Query-by-example makes data operations easy
• SODA calls are mapped to managed tables, indexes and sequences
• Production in cx_Oracle 7 with Oracle Database 18.5 or 19.3
• SODA APIs also exist in PL/SQL, Node.js, C and Java
Copyright © 2019 Oracle and/or its affiliates.
35. Creating SODA Documents
SQL> grant SODA_APP to pythondemo;
conn = cx_Oracle.connect("pythondemo/welcome@localhost/orclpdb1")
conn.autocommit = True
soda = conn.getSodaDatabase()
coll = soda.createCollection("mycollection")
content = {'name': 'Matilda', 'address': {'city': 'Melbourne'}}
coll.insertOne(content)
content = {'name': 'Mary', 'address': {'city': 'Madrid'}, 'postcode': 3207}
coll.insertOne(content)
content = {'name': 'Sally', 'address': {'city': 'Singapore'}}
doc = coll.insertOneAndGet(content)
keyForSally = doc.key
Copyright © 2019 Oracle and/or its affiliates.
36. Querying SODA Collections
doc = coll.find().key(keyForSally).getOne()
print("Document content:", doc.getContent())
print()
docs = coll.find().filter({'name': {'$like': 'Ma%'}}).getDocuments()
names = [d.getContent()["name"] for d in docs]
print("Names matching 'Ma%':", names)
print()
count = coll.find().count()
print('Collection has', count, 'documents')
Copyright © 2019 Oracle and/or its affiliates.
Document content: {'name': 'Sally', 'address': {'city': 'Singapore'}}
Names matching 'Ma%': ['Matilda', 'Mary']
Collection has 3 documents
38. Oracle Network Configuration
• sqlnet.ora and tnsnames.ora on the ‘client’
• instantclient_XX_Y/network/admin
• $TNS_ADMIN
• ~/.sqlnet.ora
• Useful sqlnet.ora options
SQLNET.OUTBOUND_CONNECT_TIMEOUT – limit the time to establish a connection
DISABLE_OOB=ON – if network drops/in-lines out-of-band breaks (auto-detected in 19c)
• Useful tnsnames.ora options (18c+)
EXPIRE_TIME – send packets for dead server detection / firewall timeout
Copyright © 2019 Oracle and/or its affiliates.
39. Oracle Client 19c Easy Connect Plus
Security, Proxy and Description parameters without config files
"mydbmachine.example.com/orclpdb1?connect_timeout=1&expire_time=3"
[[protocol:]//]host1{,host2}[:port1]{,host2:port2}
[/[service_name][:server_type]
[/instance_name]][?parameter_name=value{¶meter_name=value}]
Copyright © 2019 Oracle and/or its affiliates.
41. Oracle Database High Availability
• Premise: a database instance is always available
• RAC – Real Application Clusters
• DG – Data Guard
• ADG – Active Data Guard
• Oracle Technologies:
• Fast Application Notification (FAN)
• Application Continuity (AC)
• Transparent Application Continuity (TAC)
• . . .
Copyright © 2019 Oracle and/or its affiliates.
42. Fast Application Notification (FAN)
• Enable events mode in Python
cx_Oracle.SessionPool(..., events=True)
• Configure database service
srvctl modify service –db hr -service hasvc-notification TRUE
(RAC)
dbms_service.modify_service('hasvc', aq_ha_notfications => TRUE);
(non-RAC)
• Messages that are sent
• Down – notify apps that an instance is down
• Planned Down – notify apps of planned maintenance
• Up – notify apps that an instance is back up again
Copyright © 2019 Oracle and/or its affiliates.
43. Graceful Planned Maintenance Scenario
Copyright © 2019 Oracle and/or its affiliates.
Python app uses: cx_Oracle connection pool
DBA runs: srvctl relocate|stop service –db ... –service ... –drain_timeout ...
(not using –force)
Connections drain New work is redirected by listeners
Idle connections in the pool are released
Active connections are released when returned to pools
After drain
timeout, DBA
completes
shutdown
Wait to allow connections to drain, e.g. 10-30 minutes
exec dbms_service.disconnect_session('svcname', ...,
DBMS_SERVICE.POST_TRANSACTION); (optional)
shutdown immediate
44. Application Continuity
• On error, application automatically replays queries and
transactions to another DB instance
• Masks most hardware, software, network, storage errors and
outages
• Available with Oracle Database 12.2+
• Transparent Application Continuity with Oracle Database 18+
• RAC, RAC One, & Active Data Guard
• Enabled by a DB service setting
Copyright © 2019 Oracle and/or its affiliates.
45. Application Continuity – Two Phases
Automatic Capture
1. Pool marks start of
request
2. Client captures
original calls, their
inputs, and validation
data
3. Server decides which
can and cannot be
replayed
4. At successful end of
request, client purges
captured calls
Copyright © 2019 Oracle and/or its affiliates.
Replay on Failure
1. Client checks replay is
enabled
2. Creates a new
connection
3. Client verifies
timeliness
4. Server checks replay
database is valid
5. Server checks if
committed, rolls back
if not
Replay (cont.)
6. Client replays
captured calls
7. Server and client
ensure results
returned to
application match
original results
8. On success, client
returns control to the
application
47. • Pandas
• Jupyter Notebook
• Oracle Spatial
• OML4Py*
Copyright © 2019 Oracle and/or its affiliates.
* Coming soon
48. Time for a Demo !
Copyright © 2019 Oracle and/or its affiliates.
49. oracle.com/gbtour
New Free Tier Always Free
Oracle Cloud Infrastructure
Services you can use for unlimited time
30-Day Free Trial
Free credits you can use for more services
+