Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 51

Effective and Efficient Python with Oracle Database

4

Share

Download to read offline

These slides were presented at the AUSOUG / Oracle Groundbreakers conferences in October 2019. The abstract was "Python is rapidly becoming a de facto language for big and small database applications. Learn the best techniques for using Oracle's cx_Oracle DBI-compliant Python API and its advanced, Oracle-specific features. Whether you are creating microservices, or writing management and monitoring scripts, cx_Oracle's traditional relational data access and new document store interfaces are easy and powerful to use. This session dives into making the most of cx_Oracle, from best practice configuration for high availability, through to using the latest cx_Oracle features."

If you like this, you may want to read all the good online documentation at https://cx-oracle.readthedocs.io/en/latest/index.html

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Effective and Efficient Python with Oracle Database

  1. 1. Effective and Efficient Python with Oracle Database Christopher Jones Data Access Development Oracle Database Copyright © 2019 Oracle and/or its affiliates. christopher.jones@oracle.com @ghrd blogs..oracle.com/opal
  2. 2. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
  3. 3. Oracle Database for the Developer Copyright © 2019 Oracle and/or its affiliates. Third Party Open Source Drivers Oracle Open Source Drivers Oracle Call Interface (OCI) Typically from Oracle Instant Client ZIPs or RPMs ODPI-C Also used by custom apps Python cx_Oracle Node.js node-oracledb Go goracle Julia Oracle.jl Rust rust- oracle PHP OCI8 PHP PDO_OCI R ROracle Go go-oci8 Erlang oranif Perl DBD:: Oracle Ruby ruby- oci8 Oracle Call Interface, Oracle C++ Call Interface, ODBC, JDBC, ODP.NET, Pro*C, Pro*COBOL, SQLJ, OLE DB, OLE DB for OLAP Oracle Proprietary Drivers
  4. 4. Themes • Introduction & Installation • cx_Oracle Feature Dive • High Availability • Towards Data Science Copyright © 2019 Oracle and/or its affiliates.
  5. 5. Introduction Copyright © 2019 Oracle and/or its affiliates.
  6. 6. Python in Four Points • General purpose scripting language • Huge package library with sophisticated packages • “Everyone” is using it • Great Oracle Database (and TimesTen) support Copyright © 2019 Oracle and/or its affiliates.
  7. 7. cx_Oracle • Open Source package for Python access to Oracle Database • Covers all of Python Database API specification • Many additions to support advanced Oracle features • 50+ releases covering Oracle 8i through 19c • Begun by Anthony Tuininga in 1998 for Oracle 8i and Python 1.5 • Latest release is cx_Oracle 7.2.3 (September 2019) Copyright © 2019 Oracle and/or its affiliates.
  8. 8. cx_Oracle Features Open Source Maintained by Oracle Hosted on PyPI and GitHub Binary 'wheel' packages SQL and PL/SQL Execution Scrollable cursors Extensive data type support Binding to SQL objects, PL/SQL index-by tables, and records Fetching of large result sets REF CURSORs Copyright © 2019 Oracle and/or its affiliates. Oracle Database JSON datatype Large Objects: CLOBs and BLOBs Simple Oracle Document Access (SODA) Transaction Management Session Pooling Database Resident Connection Pooling (DRCP) Privileged Connections and Database startup/shutdown External Authentication Advanced Queuing (AQ) Continuous Query Notification (CQN) Client Result Caching End-to-end authentication and tracing Implicit Results Transaction Guard Edition Based Redefinition Row Prefetching Statement Caching Application Context Sharded Databases Oracle Database High Availability Features Oracle Net Features including Encryption
  9. 9. 10 cx_Oracle Architecture app.py cx_Oracle Oracle Call Interface in Oracle Client libraries Oracle DBPythonUsers
  10. 10. Installation in One Slide 1. Install Python 2.7 or 3.5+ 2. Install cx_Oracle: python -m pip install cx_Oracle --upgrade --user --proxy=http://proxy.example.com:80 3. Add Oracle 11.2 – 19c client libraries to the library search path • Windows: PATH • Linux: LD_LIBRARY_PATH or use ldconfig • macOS: put files in ~/lib or /usr/local/lib • Use Instant Client if your database is on a remote computer or the cloud Copyright © 2019 Oracle and/or its affiliates.
  11. 11. Ok, Two Slides 4. For Cloud DBs, download wallet.zip 5. Extract and put files in instantclient_XX_Y/network/admin cwallet.sso sqlnet.ora tnsnames.ora 6. Still need username/password at connection with cloud wallets Copyright © 2019 Oracle and/or its affiliates.
  12. 12. Connecting to the Database Copyright © 2019 Oracle and/or its affiliates.
  13. 13. Standalone Connections import cx_Oracle with cx_Oracle.connect("user", "password", "localhost/orclpdb1", encoding="UTF-8") as connection: cursor = connection.cursor() for row in cursor.execute("select * from SomeTab"): print(row) Copyright © 2019 Oracle and/or its affiliates. Python DB
  14. 14. Pooled Connections pool = cx_Oracle.SessionPool("user", "password", "localhost/orclpdb1", min=20, max=20, increment=0, threaded=True, encoding="UTF-8") . . . with pool.acquire() as connection: cursor = connection.cursor() for row in cursor.execute("select * from SomeTab"): print(row) Copyright © 2019 Oracle and/or its affiliates. Python DB
  15. 15. Many Apps Set Session State with pool.acquire() as connection: cursor = connection.cursor() cursor.callproc("set_app_state") do_work(connection) Copyright © 2019 Oracle and/or its affiliates. Round-trip required each time connection is acquired from the pool Set NLS settings, time zone, PL/SQL package state, etc. "A server round-trip is defined as the trip from the client to the database server and back to the client."
  16. 16. Session Callback: All Connections with Same State def init_session(connection, requestedTag): cursor = connection.cursor() cursor.callproc("set_app_state") pool = cx_Oracle.SessionPool("user", "password", "dsn", min=5, max=20, increment=3, sessionCallback=init_session) with pool.acquire() as connection: do_work(connection) Copyright © 2019 Oracle and/or its affiliates. Only invoked for new sessions or when the requested tag does not match the connection tag Connection returned from pool.acquire() already has the right state
  17. 17. Getting Data Out of the Database Copyright © 2019 Oracle and/or its affiliates.
  18. 18. Fetching Rows (in Batches) cursor = connection.cursor() cursor.arraysize = 500 cursor.execute("select * from SomeTab") while True: rows = cursor.fetchmany() if rows: do_something(rows) if len(rows) < cursor.arraysize: break Copyright © 2019 Oracle and/or its affiliates. Default fetchmany() value. Internal tuning buffer size for fetchone() and fetchall() too.
  19. 19. Fetching LOBs cursor = connection.cursor() cursor.execute("select ClobVal from SomeTab") for lob in cursor: lobData = lob.read() # or str(lob) Copyright © 2019 Oracle and/or its affiliates.
  20. 20. Fetching LOBs as Strings/Bytes def handler(cursor, name, defaultType, size, precision, scale): if defaultType == cx_Oracle.CLOB: return cursor.var(cx_Oracle.LONG_STRING, arraysize=cursor.arraysize) elif defaultType == cx_Oracle.BLOB: return cursor.var(cx_Oracle.LONG_BINARY, arraysize=cursor.arraysize) cursor.outputtypehandler = handler cursor.execute("select LobVal from SomeTab") for lobData in cursor: ... Copyright © 2019 Oracle and/or its affiliates.
  21. 21. Fetching Numbers as Decimals def handler(cursor, name, defaultType, size, precision, scale): if defaultType == cx_Oracle.NUMBER: return cursor.var(decimal.Decimal, arraysize=cursor.arraysize) cursor.outputtypehandler = handler for value, in cur.execute("select Data from DecimalTab"): print("Value:", value, "* 3 =", value * 3) Copyright © 2019 Oracle and/or its affiliates. Default output: Output type handler output: Value: 0.1 * 3 = 0.30000000000000004 Value: 3.1 * 3 = 9.3 Value: 7.1000000000000005 * 3 = 21.3 Value: 0.1 * 3 = 0.3 Value: 3.1 * 3 = 9.3 Value: 7.1 * 3 = 21.3
  22. 22. Putting Data Into the Database Copyright © 2019 Oracle and/or its affiliates.
  23. 23. Binding Data cursor = connection.cursor() sql = "insert into MyTab (Id, Data) values (:idVal, :dataVal)" # bind by position using a sequence (list or tuple) cursor.execute(sql, [1, "String 1"]) cursor.execute(sql, (1, "String 1")) # bind by name using a dictionary cursor.execute(sql, {"idVal": 1, "dataVal": "String 1"}) # bind by name using keyword arguments cursor.execute(sql, idVal=1, dataVal="String 1") Copyright © 2019 Oracle and/or its affiliates.
  24. 24. Batch Execution Copyright © 2019 Oracle and/or its affiliates. Execute one DML or PL/SQL statementwith many data values rows = [ (1, "First" ), (2, "Second" ), (3, "Third" ), (4, "Fourth" ), (5, "Fifth" ), (6, "Sixth" ), (7, "Seventh" ) ] cursor = connection.cursor() cursor.setinputsizes(none, 20) cursor.executemany("insert into MyTab (Id, Data) values (:idVal, :dataVal)", rows) Saves “round-trips” between Python and Oracle Database
  25. 25. Batch Errors dataToInsert = [ (1016, 10, 'Red'), (1018, 20, 'Blue'), (1018, 30, 'Green’), # duplicate key (1022, 40, 'Yellow'), (1021, 75, 'Orange'), # parent does not exist ] cursor.executemany("insert into ChildTable values (:1, :2, :3)", dataToInsert, batcherrors=True) for error in cursor.getbatcherrors(): print("Error", error.message, "at row offset", error.offset) Copyright © 2019 Oracle and/or its affiliates. Error ORA-00001: unique constraint (PYTHONDEMO.CHILDTABLE_PK) violated at row offset 2 Error ORA-02291: integrity constraint (PYTHONDEMO.CHILDTABLE_FK) violated - parent key not found at row offset 4
  26. 26. Oracle Objects and Collections Copyright © 2019 Oracle and/or its affiliates.
  27. 27. Binding Named Object Types create table TestGeometry ( IntCol number(9) not null, Geometry SDO_GEOMETRY not null ); describe SDO_GEOMETRY Name Null? Type ----------------------------------------- -------- ---------------------------- SDO_GTYPE NUMBER . . . SDO_ELEM_INFO MDSYS.SDO_ELEM_INFO_ARRAY SDO_ORDINATES MDSYS.SDO_ORDINATE_ARRAY . . . Copyright © 2019 Oracle and/or its affiliates.
  28. 28. Binding Named Object Types (continued) typeObj = conn.gettype("SDO_GEOMETRY") elementInfoTypeObj = conn.gettype("SDO_ELEM_INFO_ARRAY") ordinateTypeObj = conn.gettype("SDO_ORDINATE_ARRAY") obj = typeObj.newobject() # or typeObj() obj.SDO_GTYPE = 2003 obj.SDO_ELEM_INFO = elementInfoTypeObj.newobject() obj.SDO_ELEM_INFO.extend([1, 1003, 3]) obj.SDO_ORDINATES = ordinateTypeObj.newobject() obj.SDO_ORDINATES.extend([1, 1, 5, 7]) cur = conn.cursor() cur.execute("insert into TestGeometry values (1, :objbv)", objbv=obj) Copyright © 2019 Oracle and/or its affiliates.
  29. 29. Fetching Objects def dumpobject(obj, prefix = " "): if obj.type.iscollection: print(prefix, "[") for value in obj.aslist(): if isinstance(value, cx_Oracle.Object): dumpobject(value, prefix + " ") else: print(prefix + " ", repr(value)) print(prefix, "]") else: print(prefix, "{") for attr in obj.type.attributes: value = getattr(obj, attr.name) if isinstance(value, cx_Oracle.Object): print(prefix + " " + attr.name + " :") dumpobject(value, prefix + " ") else: print(prefix + " " + attr.name + " :", repr(value)) print(prefix, "}") Copyright © 2019 Oracle and/or its affiliates.
  30. 30. Advanced Queuing Copyright © 2019 Oracle and/or its affiliates.
  31. 31. Enqueuing SQL> call dbms_aqadm.create_queue_table('RAW_QUEUE_TAB', 'RAW’); SQL> call dbms_aqadm.create_queue('RAW_QUEUE', 'RAW_QUEUE_TAB’); SQL> call dbms_aqadm.start_queue('RAW_QUEUE'); data = [ "Message 1", "Message 2", ... ] queue = connection.queue("RAW_QUEUE") # enqueue one at a time for payloadData in data: queue.enqOne(connection.msgproperties(payload=payloadData)) # enqueue many at a time messages = [connection.msgproperties(payload=d) for d in data] queue.enqMany(messages) Copyright © 2019 Oracle and/or its affiliates.
  32. 32. Dequeuing queue = connection.queue("RAW_QUEUE") # dequeue one at a time message = queue.deqOne() print(message.payload.decode(connection.encoding)) # dequeue many at a time maxNumMessages = 8 for message in queue.deqMany(maxNumMessages): print(message.payload.decode(connection.encoding)) Copyright © 2019 Oracle and/or its affiliates.
  33. 33. SODA Copyright © 2019 Oracle and/or its affiliates.
  34. 34. Simple Oracle Document Access (SODA) • NoSQL-style APIs to create and access documents • Documents are often JSON • Query-by-example makes data operations easy • SODA calls are mapped to managed tables, indexes and sequences • Production in cx_Oracle 7 with Oracle Database 18.5 or 19.3 • SODA APIs also exist in PL/SQL, Node.js, C and Java Copyright © 2019 Oracle and/or its affiliates.
  35. 35. Creating SODA Documents SQL> grant SODA_APP to pythondemo; conn = cx_Oracle.connect("pythondemo/welcome@localhost/orclpdb1") conn.autocommit = True soda = conn.getSodaDatabase() coll = soda.createCollection("mycollection") content = {'name': 'Matilda', 'address': {'city': 'Melbourne'}} coll.insertOne(content) content = {'name': 'Mary', 'address': {'city': 'Madrid'}, 'postcode': 3207} coll.insertOne(content) content = {'name': 'Sally', 'address': {'city': 'Singapore'}} doc = coll.insertOneAndGet(content) keyForSally = doc.key Copyright © 2019 Oracle and/or its affiliates.
  36. 36. Querying SODA Collections doc = coll.find().key(keyForSally).getOne() print("Document content:", doc.getContent()) print() docs = coll.find().filter({'name': {'$like': 'Ma%'}}).getDocuments() names = [d.getContent()["name"] for d in docs] print("Names matching 'Ma%':", names) print() count = coll.find().count() print('Collection has', count, 'documents') Copyright © 2019 Oracle and/or its affiliates. Document content: {'name': 'Sally', 'address': {'city': 'Singapore'}} Names matching 'Ma%': ['Matilda', 'Mary'] Collection has 3 documents
  37. 37. High Availability Copyright © 2019 Oracle and/or its affiliates.
  38. 38. Oracle Network Configuration • sqlnet.ora and tnsnames.ora on the ‘client’ • instantclient_XX_Y/network/admin • $TNS_ADMIN • ~/.sqlnet.ora • Useful sqlnet.ora options SQLNET.OUTBOUND_CONNECT_TIMEOUT – limit the time to establish a connection DISABLE_OOB=ON – if network drops/in-lines out-of-band breaks (auto-detected in 19c) • Useful tnsnames.ora options (18c+) EXPIRE_TIME – send packets for dead server detection / firewall timeout Copyright © 2019 Oracle and/or its affiliates.
  39. 39. Oracle Client 19c Easy Connect Plus Security, Proxy and Description parameters without config files "mydbmachine.example.com/orclpdb1?connect_timeout=1&expire_time=3" [[protocol:]//]host1{,host2}[:port1]{,host2:port2} [/[service_name][:server_type] [/instance_name]][?parameter_name=value{&parameter_name=value}] Copyright © 2019 Oracle and/or its affiliates.
  40. 40. Execution Timeout cx_Oracle Connection.callTimeout • milliseconds any round-trip to the DB may take • returns error to the app Copyright © 2019 Oracle and/or its affiliates.
  41. 41. Oracle Database High Availability • Premise: a database instance is always available • RAC – Real Application Clusters • DG – Data Guard • ADG – Active Data Guard • Oracle Technologies: • Fast Application Notification (FAN) • Application Continuity (AC) • Transparent Application Continuity (TAC) • . . . Copyright © 2019 Oracle and/or its affiliates.
  42. 42. Fast Application Notification (FAN) • Enable events mode in Python cx_Oracle.SessionPool(..., events=True) • Configure database service srvctl modify service –db hr -service hasvc-notification TRUE (RAC) dbms_service.modify_service('hasvc', aq_ha_notfications => TRUE); (non-RAC) • Messages that are sent • Down – notify apps that an instance is down • Planned Down – notify apps of planned maintenance • Up – notify apps that an instance is back up again Copyright © 2019 Oracle and/or its affiliates.
  43. 43. Graceful Planned Maintenance Scenario Copyright © 2019 Oracle and/or its affiliates. Python app uses: cx_Oracle connection pool DBA runs: srvctl relocate|stop service –db ... –service ... –drain_timeout ... (not using –force) Connections drain New work is redirected by listeners Idle connections in the pool are released Active connections are released when returned to pools After drain timeout, DBA completes shutdown Wait to allow connections to drain, e.g. 10-30 minutes exec dbms_service.disconnect_session('svcname', ..., DBMS_SERVICE.POST_TRANSACTION); (optional) shutdown immediate
  44. 44. Application Continuity • On error, application automatically replays queries and transactions to another DB instance • Masks most hardware, software, network, storage errors and outages • Available with Oracle Database 12.2+ • Transparent Application Continuity with Oracle Database 18+ • RAC, RAC One, & Active Data Guard • Enabled by a DB service setting Copyright © 2019 Oracle and/or its affiliates.
  45. 45. Application Continuity – Two Phases Automatic Capture 1. Pool marks start of request 2. Client captures original calls, their inputs, and validation data 3. Server decides which can and cannot be replayed 4. At successful end of request, client purges captured calls Copyright © 2019 Oracle and/or its affiliates. Replay on Failure 1. Client checks replay is enabled 2. Creates a new connection 3. Client verifies timeliness 4. Server checks replay database is valid 5. Server checks if committed, rolls back if not Replay (cont.) 6. Client replays captured calls 7. Server and client ensure results returned to application match original results 8. On success, client returns control to the application
  46. 46. Towards Data Science Copyright © 2019 Oracle and/or its affiliates.
  47. 47. • Pandas • Jupyter Notebook • Oracle Spatial • OML4Py* Copyright © 2019 Oracle and/or its affiliates. * Coming soon
  48. 48. Time for a Demo ! Copyright © 2019 Oracle and/or its affiliates.
  49. 49. oracle.com/gbtour New Free Tier Always Free Oracle Cloud Infrastructure Services you can use for unlimited time 30-Day Free Trial Free credits you can use for more services +
  50. 50. Resources Mail: christopher.jones@oracle.com Twitter: @ghrd Blog: blogs.oracle.com/opal Home: https://oracle.github.io/python-cx_Oracle Always Free: oracle.com/gbtour
  51. 51. Thank You Christopher Jones christopher.jones@oracle.com Data Access Development Oracle Database

×