DevoxxFR 2024 Reproducible Builds with Apache Maven
Automate That! Scripting Atlassian applications in Python
1.
2. Has this happened to you? Email to users results in 50+ undeliverable Need to verify the users in Active Directory Then “deactivate” former employees in Crowd 750 mouse clicks later, you’re done! 2 http://www.flickr.com/photos/left-hand/4231405740/
4. Agenda Use cases for scripting Atlassian APIs available for scripting The awesome power and simplicity of python Examples 4
5. When is scripting useful? Automate time consuming tasks Perform data analysis Cross-reference data from multiple systems 5
6. Some specific use cases Crowd – Deactivate Users and remove from all groups Bamboo – Disable all plans in a project JIRA – Release Notes Subversion – custom commit acceptance Custom build processes – pull code linked to a specific issue into a patch archive 6
12. More APIs for scripting(the ones we prefer to use) RESTful Remote APIs (now deprecated) High level interface Supports a handful of actions Now emerging: “real” REST interfaces High level interface Supports a handful of actions http://confluence.atlassian.com/display/REST/Guidelines+for+Atlassian+REST+API+Design 9
13. Why Python? Powerful standard libraries Http(s) with cookie handling XML and JSON Unicode Third Party Libraries SOAP REST Templates Subversion Portable, cross-platform 10
14. Python Versions 2.x Ships with most linux distributions Lots of third-party packages available 3.x Latest version Deliberately incompatible with 2.x Not as many third-party libraries 11
15. HTTP(s) with Python Python 2 httplib – low level, all HTTP verbs urllib – GET and POST, utilities urllib2 – GET and POST using Request class, easier manipulation of headers, handlers for cookies, proxies, etc. Python 3 http.client – low level, all HTTP verbs http.parse - utilities urllib.request – similar to urllib2 Third-Party httplib2 – high-level interface with all HTTP verbs, plus caching, compression, etc. 12
18. Simple Issue Retrieval 15 import urllib, httplib import xml.etree.ElementTree as etree jira_serverurl = 'http://jira.atlassian.com' jira_userid = 'myuserid' jira_password = 'mypassword' detailsURL = jira_serverurl + br /> "/si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml" + br /> "?os_username=" + jira_userid + "&os_password=" + jira_password f = urllib.urlopen(detailsURL) tree=etree.parse(f) f.close() Construct a URL that looks like the one in the UI, with extra parms for our user auth Open the URL with one line! Parse the XML with one line!
19. Find details in XML 16 Find based on tag name or path to element details = tree.getroot() print "Issue: " + details.find("channel/item/key").text print "Status: " + details.find("channel/item/status").text print "Summary: " + details.find("channel/item/summary").text print "Description: " + details.find("channel/item/description").text Issue: JRA-9 Status: Open Summary: User Preference: User Time Zones Description: <p>Add time zones to user profile. That way the dates displayed to a user are always contiguous with their local time zone, rather than the server's time zone.</p>
20. Behind the scenes…cookies! 17 Turn on debugging and see exactly what’s happening httplib.HTTPConnection.debuglevel= 1 f = urllib.urlopen(detailsURL) send: 'GET /si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml?os_username=myuserid&os_password=mypassword HTTP/1.0Host: jira.atlassian.comUser-Agent: Python-urllib/1.17' reply: 'HTTP/1.1 200 OK' header: Date: Wed, 20 Apr 2011 12:04:37 GMT header: Server: Apache-Coyote/1.1 header: X-AREQUESTID: 424x2804517x1 header: X-Seraph-LoginReason: OK header: X-AUSERNAME: myuserid header: X-ASESSIONID: 19b3b8o header: Content-Type: text/xml;charset=UTF-8 header: Set-Cookie: JSESSIONID=A1357C4805B1345356404A65333436D3; Path=/ header: Set-Cookie: atlassian.xsrf.token=AKVY-YUFR-9LM7-97AB|e5545d754a98ea0e54f 8434fde36326fb340e8b7|lin; Path=/ header: Connection: close JSESSIONID cookie sent from JIRA
21. Authentication User credentials determine: The data returned The operations allowed Methods Available: Basic Authentication JSESSIONID Cookie Token Method 18
23. JSESSIONID Cookie Authentication credentials passed once; then cookie is used Used when scripting the user interface Can be used with REST API for JIRA, Confluence, and Bamboo 20
24. Token Method Authentication credentials passed once; then token is used Used with Fisheye/Crucible REST Used with Deprecated Bamboo Remote API 21
25. Obtaining a cookie Scripting the user interface login page Adding parameters to the user interface URL: “?os_username=myUserID&os_password=myPassword” Using the JIRA REST API 22
26. JIRA REST Authentication 23 import urllib, urllib2, cookielib, json # set up cookiejar for handling URLs cookiejar = cookielib.CookieJar() myopener= urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar)) creds = { "username" : jira_userid, "password" : jira_password } queryurl = jira_serverurl + "/rest/auth/latest/session" req = urllib2.Request(queryurl) req.add_data(json.dumps(creds)) req.add_header("Content-type", "application/json") req.add_header("Accept", "application/json") fp= myopener.open(req) fp.close() urllib2 handles cookies automatically. We just need to give it a CookieJar Request and response are both JSON We don’t care about response, just the cookie
27. Submitting a JIRA Querywith the user interface 24 # Search using JQL queryJQL = urllib.quote("key in watchedIssues()") queryURL = jira_serverurl + br /> "/sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml" + br /> "?tempMax=1000&jqlQuery=" + queryJQL fp = myopener.open(queryURL) # Search using an existing filter filterId = "20124" queryURL = jira_serverurl + br /> "/sr/jira.issueviews:searchrequest-xml/" + br /> "{0}/SearchRequest-{0}.xml?tempMax=1000".format(filterId) fp = myopener.open(queryURL) Pass any JQL Query Or Pass the ID of an existing shared filter
28. A JQL Query using REST 25 # Search using JQL queryJQL= "key in watchedIssues()" IssuesQuery= { "jql" : queryJQL, "startAt" : 0, "maxResults" : 1000 } queryURL = jira_serverurl + "/rest/api/latest/search" req = urllib2.Request(queryURL) req.add_data(json.dumps(IssuesQuery)) req.add_header("Content-type", "application/json") req.add_header("Accept", "application/json") fp= myopener.open(req) data = json.load(fp) fp.close() Pass any JQL Query Request and response are both JSON
29. XML returned from user interface query 26 An RSS Feed with all issues and requested fields that have values
30. JSON returnedfrom a REST query 27 {u'total': 83, u'startAt': 0, u'issues': [{u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23969', u'key': u'JRA-23969'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23138', u'key': u'JRA-23138'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2770', u'key': u'BAM-2770'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2489', u'key': u'BAM-2489'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1410', u'key': u'BAM-1410'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1143', u'key': u'BAM-1143'}], u'maxResults': 200} A list of the issues found, with links to retrieve more information
31. JSON issue details 28 All applicable fields are returned, even if there’s no value Expand the html property to get rendered html for description, comments
41. Which build resolved my issue? Bamboo keeps track of “related issues” (based on issue IDs included in commit comments), but doesn’t know when issues are resolved. If we know the issue is resolved in JIRA, we can look to see the latest build that lists our ID as a “related issue” Not a continuous integration build? We’ll need to look in fisheye to determine the highest revision related to this issue and then look in bamboo to see if a build using this revision has completed successfully. 32
42. To Fisheye for related commits! 33 queryURL= FisheyeServer + "/rest-service-fe/changeset-v1/listChangesets" + br /> "?rep={0}&comment={1}&expand=changesets".format(FisheyeRepo, myissue) req= urllib2.Request(queryURL) auth_string = '{0}:{1}'.format(fisheye_userid,fisheye_password) base64string = base64.encodestring(auth_string)[:-1] req.add_header("Authorization", "Basic {0}".format(base64string)) response = myopener.open(req) issuecommits=etree.parse(response).getroot() response.close() Query a specific fisheye repository for a commit with our JIRA issue ID in the comments Use basic auth headers to authenticate
44. Parsing the changesets 35 commits = [] for changeset in issuecommits.findall("changesets/changeset"): commits.append(changeset.findtext("csid")) commits.sort() print "Highest commit is: " + commits[-1] Highest commit is: 130948
45. Logging into Bamboo 36 urllib2.HTTPCookieProcessor(cookiejar)) cookiejar = cookielib.CookieJar() myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar)) queryURL = bambooServer + "/userlogin!default.action“ params= urllib.urlencode({ "os_username" : bambooUserid, "os_password" : bambooPassword}) response = myopener.open(queryURL, params) response.close() Using a POST to the user interface login screen to retrieve a JSESSIONID cookie
46. Querying for build results 37 # Warning: This is a very resource-intensive operation. # You should consider limiting the number of builds returned queryURL= bambooServer + "/rest/api/latest/result/MYPROJECT-MYPLAN" + br /> "?expand=results[-10:-1].result.jiraIssues" req = urllib2.Request(queryURL) req.add_header("Accept", "application/xml") response = myopener.open(req) results=etree.parse(response).getroot() response.close() Use negative indexes to return the last entries in build list, e.g. [-10:-1] returns last ten builds in list Request the related issues Ask for XML (JSON also available)
47. Example (partial) build results 38 <results expand="results"> <link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN" rel="self" /> <results expand="result" max-result="25" size="46" start-index="0"> <result expand="comments,labels,jiraIssues,stages" id="3146125" key="MYPROJECT-MYPLAN-26" lifeCycleState="Finished" number="26" state="Successful"> <link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN-26" rel="self" /> <buildStartedTime>2011-04-29T05:04:14.460-05:00</buildStartedTime> <buildCompletedTime>2011-04-29T05:34:35.687-05:00</buildCompletedTime> <buildRelativeTime>4 days ago</buildRelativeTime> <vcsRevisionKey>4483</vcsRevisionKey> <buildReason>Code has changed</buildReason> <comments max-result="0" size="0" start-index="0" /> <labels max-result="0" size="0" start-index="0" /> <jiraIssues max-result="1" size="1" start-index="0"> <issue iconUrl="http://myjira.domain.com/images/icons/bug.gif" issueType="Defect" key="MYJIRAPROJECT-1629" summary="Need to display an error message when balance is zero."> <urlhref="http://myjira.domain.com/browse/MYJIRAPROJECT-1629" rel="self" /> </issue> </jiraIssues> <stages max-result="1" size="1" start-index="0" /> </result> </results> </results> Can also expand comments, labels, and stages jiraIssues property has been expanded here
48. Walking through build results 39 for result in results.findall("results/result"): print result.get("key") + ":" print "Revision: " + result.findtext("vcsRevisionKey") issues = [issue.get("key") for issue in result.findall("jiraIssues/issue")] print "Issues: " + ", ".join(issues) MYPROJECT-MYPLAN-31: Revision: 4489 Issues: MYJIRAPROJECT-1658 MYPROJECT-MYPLAN-30: Revision: 4486 Issues: MYJIRAPROJECT-1630 MYPROJECT-MYPLAN-29: Revision: 4485 Issues: MYJIRAPROJECT-1616, MYJIRAPROJECT-1663
50. Beyond GET and POST 41 Lower-level HTTPConnection needed connection = httplib.HTTPConnection('myCrowdServer.mydomain.com:8080') operation = 'DELETE' urlpath = "/rest/usermanagement/latest/user/group/direct" + br /> "?username={0}&groupname={1}".format(userToRemove, fromGroup) body = None auth_string = '{0}:{1}'.format(crowdAppName,crowdAppPassword) base64string = base64.encodestring(auth_string)[:-1] headers = {'Authorization' : "Basic {0}".format(base64string)} connection.request(operation, urlpath, body, headers) response = connection.getresponse() print response.status, response.reason connection.close() Authenticate as a Crowd Application 204 - group membership is successfully deleted 403 - not allowed to delete the group membership 404 - the user or group or membership could not be found
51. A few loose ends Be prepared to handle Unicode strings Error handling – not shown here, but important! Formatting output – several python libraries for handling templates are available REST Interfaces – you can write your own! http://confluence.atlassian.com/display/DEVNET/Plugin+Tutorial+-+Writing+REST+Services 42
52. Links for more information http://confluence.atlassian.com/display/JIRA/Displaying+Search+Results+in+XML http://confluence.atlassian.com/display/JIRA/JIRA+REST+API+(Alpha)+Tutorial http://confluence.atlassian.com/display/CONFDEV/Confluence+REST+APIs http://confluence.atlassian.com/display/FECRUDEV/REST+API+Guide http://confluence.atlassian.com/display/BAMBOO/Bamboo+REST+APIs http://confluence.atlassian.com/display/CROWDDEV/Crowd+REST+APIs 43
Notes de l'éditeur
FIS is part of the S&P 500 and is one of the world's top-ranked technology providers to the banking industry.
Three basic scenarios where scripting is useful
JIRA – JQL provides amazing ability to search for issues. The presentation choices are limited, however, particularly if you want a report that you can email to others.
We have 2,000 users, so we tend to value server stability
All examples here in python 2
No proper error handling in any of these examples
Note that fields with wiki markup are returned as-is
Here, we’re going to use cookies. The JSESSIONID cookie can be used interchangeably between REST and non-REST callsYou can also use basic auth
No fields specified, so all fields are returned for all issues
Currently not possible to search based on an existing filter using REST
No fields are returned…. Just a list of the issues
Custom fields are listed alongside system fields
The contents of the “value” are highly dependent on the field type
The revisions “size” attribute tells us how many files were committed in that changeset
In practice, it seems that changesets are returned in decreasing order. The documentation doesn’t specify any order, however, so we’ll make no assumptions here
Bamboo allows basic auth, but you have to supply “os_authType=basic” as a query parameter in addition to the basicauth header. Here, we elect to exercise the user interface and obtain a cookie instead.
Just asking for the last 10 build results – would really want to loop backwards a chunk at a timeExpanding the related jira issues – can expand other fields as well Requesting the results in xml. Json also available
Note that we can also expand comment, labels, and stages (in addition to related issues)