Why would a company release an open source python plugin with limited features and hide away the full features behind a pay wall, when the code is straight forward and easily extensible. Answer: Because the implementation of all available API’s is tedious. That’s the lesson I learned when expanding the features of the python-sonarqube-api project.
Implementation
Earlier this year we setup a SonarQube site to track build tests and set quality gates for pull requests in order to maintain code quality. Through the course of enabling the various projects it was determined that we also needed to be able to dynamically modify the projects within a set of management pipelines. SonarQube provides several API endpoints in order to fulfill these demands. In Bash it is easy enough to access these endpoints using Curl. However the pipelines in question ran a python script for all it’s actions.
The python alternative to Curl is requests and my initial attempts at accessing the endpoints with requests were a failure. Not finding and adequate solution in my initial search engine queries. I did however find the plugin python-sonarqube-api. After adding the plugin to the required.txt files and writing the necessary code, I was immediately disappointed to find that the API calls that I needed to use were held back from the free version and was only available in a paid plugin. However sifting through the source code of the free plugin and further search engine queries uncovered exactly what the plugin was doing. Enough for me to implement my own solution.
There are categories for each function an API address broken down as follows. api/<function>/<action>. These endpoints are stored in sonarqube/utils/config.py.
API_PROJECTS_SEARCH_ENDPOINT = "/api/projects/search"class SonarQubeDuplications(RestClient):
@GET(API_DUPLICATIONS_SHOW_ENDPOINT)
def get_duplications(self, key, branch=None, pullRequest=None):In this file endpoints are assigned to variables as demonstrated in the first example above. Within the folder sonarqube/rest/<function>.py each API endpoint has a wrapper of either GET or POST and a function that will house all the possible values excepted by the API. Optional values will be accompanied by “=None” so that those values can be omitted when writting your code.
def GET(url_pattern):
"""
:param url_pattern:
:return:
"""
return endpoint(url_pattern, method="GET")The above code is the wrapper which is located in the sonarqube/utils/common.py. The function calls another function located in the same file that unpacks the wrapped function extracts the values from the arguments of the function and stores them in a dictionary. It is not necessary to demonstrate that portion of the code as it is not that important. It is called on line 11 and 12.
def wrapped_func(f):
@wraps(f)
def inner_func(self, *args, **kwargs):
"""
:param self:
:param args:
:param kwargs:
:return:
"""
func_params = translate_params(f, *args, **kwargs)
params = translate_special_params(func_params, self.special_attributes_map)
response = None
if method == "GET":
response = self._get(url_pattern, params=params)
elif method == "POST":
response = self._post(url_pattern, data=params)Finally based on the method variable runs either the self._get of self._post functions. These functions are declared in the sonarqube/utils/rest_client.py. In the sonarqube/rest/__init__.py file the requests module is initiated and assigned to self.session and self is passed along in the variable api as shown below.
import requests
from requests.auth import HTTPBasicAuth
def __init__(
self,
sonarqube_url=None,
username=None,
password=None,
token=None,
verify=None,
cert=None,
timeout=None
):
self.base_url = strip_trailing_slash(sonarqube_url or self.DEFAULT_URL)
_session = requests.Session()
self.session = _session @property
def duplications(self):
"""
SonarQube duplications Operations
:return:
"""
return SonarQubeDuplications(api=self) def _get(self, path, data=None, params=None, headers=None):
"""
Get request based on the python-requests module.
:param path:
:param data:
:param params:
:param headers:
:return:
"""
return self.request("GET", path=path, params=params, data=data, headers=headers) def __init__(self, api):
self.api = api
def request(
self,
method="GET",
path="/",
data=None,
json=None,
params=None,
headers=None,
files=None,
):
url = self.url_joiner(self.api.base_url, path)
timeout = self.api.timeout or self.default_timeout
res = self.api.session.request(
method=method,
url=url,
params=params,
headers=headers,
data=data,
json=json,
files=files,
timeout=timeout,
)
res.encoding = "utf-8"And here we see everything finally come together. In line 2 of the above code snippet the api value is assigned to self.api. On line 19 a session request is made to the api endpoint along with all the relevant data.
After reading several posts from stack overflow I realized the reason for the requests failure. The SonarQube API’s requires a session to be established in order to interact with the endpoints. Below is the all the code necessary to implement API functionality in the scripts for the projects.
The data being the relevant key value pairs for the given endpoint being accessed.
import requests
from requests.auth import HTTPBasicAuth
session = requests.Session()
auth = HTTPBasicAuth(<token>, "")
session.auth=auth
res = session.post(<endpoint>, data=<data>)Forking
For as short as this section will be. It does not in anyway convey the tedium involved in the implementation. To extend the functionality of this plugin we must first assign the endpoints to a variable in the sonarqube/utils/config.py file. Then we create a class in the sonarqube/rest/<function>.py file my example being sonarqube/rest/duplication.py.
If a new file needs to be created then we add a function to return the class of that file in the sonarqube/rest/__init__.py file. I initially believed adding the missing features to this plugin would take no more than a weekend to implement. Two weeks later, with a lengthy break in between, I can finally say I am complete, ish. All of the functions storing the key value pairs for the API have lengthy explanations detailing what the values are and version information. Though I didn’t update any comments. I did leave space for them within the code.
Conclusion
To be honest, I would have never went forward with this had I not thought documenting the process would be a good article for my blog. I’m glad I was able to complete it and that it was easy enough to understand. My first attempt to fork a project was an attempt to implement Bcrypt into LDAP. Unfortunately due to my lack of experience and lack of familiarity with C, I ultimately failed.
But I have learned you never really fail if you learn from your experiences. One of the greatest assets that has enabled me to be a good DevOps Engineer is my ability to read through a code base. I would not have developed such a skill had I not tried to fork projects in the past. This skill has helped me in troubleshooting issues in my pipelines, configurations and builds. Even allowing me to pass along the exact problem and the location to developers, of the code that resulted in a failed build or deployment.
I encourage anyone out there to at least attempt to fork a opensource project and expand the features for yourself. The skills you gain will end up having immeasurable benefits for you in your growing career.
Link to the forked project: https://github.com/newknowledg/python-sonarqube-api
