Classify Jira Tickets with GenAI On Amazon Bedrock | by Tanner McRae | Nov, 2024

Keep in mind the times when classifying textual content meant embarking on a machine studying journey? When you’ve been within the ML area lengthy sufficient, you’ve most likely witnessed not less than one group disappear down the rabbit gap of constructing the “excellent” textual content classification system. The story often goes one thing like this:

• Month 1: “We’ll simply rapidly prepare a NLP mannequin!”
• Month 2: “We want extra coaching knowledge…”
• Month 3: “That is ok”

For years, textual content classification has fallen into the realm of traditional ML. Early in my profession, I keep in mind coaching a assist vector machine (SVM) for e mail classification. Numerous preprocessing, iteration, knowledge assortment, and labeling.

However right here’s the twist: it’s 2024, and generative AI fashions can “typically” classify textual content out of the field! You’ll be able to construct a sturdy ticket classification system with out, gathering 1000’s of labeled coaching examples, managing ML coaching pipelines, or sustaining customized fashions.

On this put up, we’ll go over easy methods to setup a Jira ticket classification system utilizing giant language fashions on Amazon Bedrock and different AWS providers.

DISCLAIMER: I’m a GenAI Architect at AWS and my opinions are my very own.

Why Classify Jira Tickets?

A standard ask from corporations is to grasp how groups spend their time. Jira has tagging options, however it may possibly typically fall brief by way of human error or lack of granularity. By doing this train, organizations can get higher insights into their group actions, enabling data-driven selections about useful resource allocation, mission funding, and deprecation.

Why Not Use Different NLP Approaches?

Conventional ML fashions and smaller transformers like BERT want a whole bunch (or 1000’s) of labeled examples, whereas LLMs can classify textual content out of the field. In our Jira ticket classification checks, a prompt-engineering method matched or beat conventional ML fashions, processing 10k+ annual tickets for ~$10/12 months utilizing Claude Haiku (excluding different AWS Service prices). Additionally, prompts are simpler to replace than retraining fashions.

This github repo incorporates a pattern software that connects to Jira Cloud, classifies tickets, and outputs them in a format that may be consumed by your favourite dashboarding software (Tableu, Quicksight, or another software that helps CSVs).

Necessary Discover: This mission deploys assets in your AWS atmosphere utilizing Terraform. You’ll incur prices for the AWS assets used. Please concentrate on the pricing for providers like Lambda, Bedrock, Glue, and S3 in your AWS area.

Pre Requisites

You’ll have to have terraform put in and the AWS CLI put in within the atmosphere you wish to deploy this code from

The structure is fairly straight ahead. Yow will discover particulars beneath.

Step 1: An AWS Lambda perform is triggered on a cron job to fetch jira tickets based mostly on a time window. These tickets are then formatted and pushed to an S3 bucket below the /unprocessed prefix.

Step 2: A Glue job is triggered off /unprocessed object places. This runs a PySpark deduplication job to make sure no duplicate tickets make their method to the dashboard. The deduplicated tickets are then put to the /staged prefix. That is helpful for circumstances the place you manually add tickets in addition to depend on the automated fetch. When you can guarantee no duplicates, you may take away this step.

Step 3: A classification job is kicked off on the brand new tickets by calling Amazon Bedrock to categorise the tickets based mostly on a immediate to a big language mannequin (LLM). After classification, the completed outcomes are pushed to the /processed prefix. From right here, you may decide up the processed CSV utilizing any dashboarding software you’d like that may eat a CSV.

To get began, clone the github repo above and transfer to the /terraform listing

$ git clone https://github.com/aws-samples/jira-ticket-classification.git
$ cd jira-ticket-classification/terraform

Run terraform init, plan, & apply. Be sure you have terraform put in in your laptop and the AWS CLI configured.

$ terraform init
$ terraform plan
$ terraform apply

As soon as the infrastructure is deployed into your account, you may navigate to AWS Secrets and techniques Supervisor and replace the key along with your Jira Cloud credentials. You’ll want an API key, base url, and e mail to allow the automated pull

And that’s it!

You’ll be able to (1) look ahead to the Cron to kick off an automated fetch, (2) export the tickets to CSV and add them to the /unprocessed S3 bucket prefix, or (3) manually set off the Lambda perform utilizing a take a look at.

Jira Fetch:

Jira fetch makes use of a Lambda perform with a Cloudwatch cron occasion to set off it. The Lambda pulls within the AWS Secret and makes use of a get request shortly loop to retrieve paginated outcomes till the JQL question completes:

def fetch_jira_issues(base_url, project_id, e mail, api_key):
url = f"{base_url}/relaxation/api/3/search"
# Calculate the date 8 days in the past
eight_days_ago = (datetime.now() - timedelta(days=8)).strftime("%Y-%m-%d")
# Create JQL
jql = f"mission = {project_id} AND created >= '{eight_days_ago}' ORDER BY created DESC"
# Cross into params of request.
params = {
"jql": jql,
"startAt": 0
}
all_issues = []
auth = HTTPBasicAuth(e mail, api_key)
headers = {"Settle for": "software/json"}
whereas True:
response = requests.get(url, headers=headers, params=params, auth=auth)
if response.status_code != 200:
elevate Exception(f"Did not fetch points for mission {project_id}: {response.textual content}")
knowledge = json.hundreds(response.textual content)
points = knowledge['issues']
all_issues.lengthen(points)
if len(all_issues) >= knowledge['total']:
break
params['startAt'] = len(all_issues)
return all_issues

It then creates a string illustration of a CSV and uploads it into S3:

def upload_to_s3(csv_string, bucket, key):
strive:
s3_client.put_object(
Bucket=bucket,
Key=key,
Physique=csv_string,
ContentType='textual content/csv'
)
besides Exception as e:
elevate Exception(f"Did not add CSV to S3: {str(e)}")

Glue Job

An S3 occasion on the /unprocessed prefix kicks off a second lambda that begins an AWS Glue job. That is helpful when there’s a number of entry factors that Jira tickets can enter the system by way of. For instance, if you wish to do a backfill.

import boto3 
# Initialize Boto3 Glue shopper
glue_client = boto3.shopper('glue')
def handler(occasion, context):
# Print occasion for debugging
print(f"Acquired occasion: {json.dumps(occasion)}")
# Get bucket title and object key (file title) from the S3 occasion
strive:
s3_event = occasion['Records'][0]['s3']
s3_bucket = s3_event['bucket']['name']
s3_key = s3_event['object']['key']
besides KeyError as e:
print(f"Error parsing S3 occasion: {str(e)}")
elevate
response = glue_client.start_job_run(
JobName=glue_job_name,
Arguments={
'--S3_BUCKET': s3_bucket,
'--NEW_CSV_FILE': s3_key
}
)

The Glue job itself is written in PySpark and could be discovered within the code repo right here. The essential take away is that it does a leftanti be a part of utilizing the difficulty Ids on the gadgets within the new CSV towards all of the Ids within the /staged CSVs.

The outcomes are then pushed to the /staged prefix.

Classify Jira Tickets:

That is the place it it will get fascinating. Because it seems, utilizing immediate engineering can carry out on par, if not higher, than a textual content classification mannequin utilizing a pair strategies.

• You’ll be able to outline the classifications and their descriptions in a immediate,
• Ask the mannequin to assume step-by-step (Chain of Thought).
• After which output the classification with out having to coach a single mannequin. See the immediate beneath:

Notice: It’s essential to validate your immediate utilizing a human curated subset of labeled / labelled tickets. It’s best to run this immediate by way of the validation dataset to ensure it aligns with the way you count on the tickets to be labeled

SYSTEM_PROMPT = '''
You're a assist ticket assistant. You're given fields of a Jira ticket and your job is to categorise the ticket based mostly on these fields
Beneath is the record of potential classifications together with descriptions of these classifications.
<classifications>
ACCESS_PERMISSIONS_REQUEST: Used when somebody would not have the write permissions or cannot log in to one thing or they can not get the right IAM credentials to make a service work.
BUG_FIXING: Used when one thing is failing or a bug is discovered. Usually occasions the descriptions embrace logs or technical info.
CREATING_UPDATING_OR_DEPRECATING_DOCUMENTATION: Used when documentation is outdated. Often references documentation within the textual content.
MINOR_REQUEST: That is hardly ever used. Often a bug repair nevertheless it's very minor. If it appears even remotely difficult use BUG_FIXING.
SUPPORT_TROUBLESHOOTING: Used when asking for assist for some engineering occasion. May appear like an automatic ticket.
NEW_FEATURE_WORK: Often describes a brand new characteristic ask or one thing that is not operational.
</classifications>
The fields accessible and their descriptions are beneath.
<fields>
Summmary: This can be a abstract or title of the ticket
Description: The outline of the difficulty in pure language. Nearly all of context wanted to categorise the textual content will come from this subject
</fields>
<guidelines>
* It's doable that some fields could also be empty wherein case ignore them when classifying the ticket
* Assume by way of your reasoning earlier than making the classification and place your thought course of in <considering></considering> tags. That is your area to assume and cause in regards to the ticket classificaiton.
* After getting completed considering, classify the ticket utilizing ONLY the classifications listed above and place it in <reply></reply> tags.
</guidelines>'''
USER_PROMPT = '''
Utilizing solely the ticket fields beneath:
<summary_field>
{abstract}
</summary_field>
<description_field>
{description}
</description_field>
Classify the ticket utilizing ONLY 1 of the classifications listed within the system immediate. Keep in mind to assume step-by-step earlier than classifying the ticket and place your ideas in <considering></considering> tags.
If you end up completed considering, classify the ticket and place your reply in <reply></reply> tags. ONLY place the classifaction within the reply tags. Nothing else.
'''

We’ve added a helper class that threads the calls to Bedrock to hurry issues up:

import boto3
from concurrent.futures import ThreadPoolExecutor, as_completed
import re
from typing import Listing, Dict
from prompts import USER_PROMPT, SYSTEM_PROMPT
class TicketClassifier:
SONNET_ID = "anthropic.claude-3-sonnet-20240229-v1:0"
HAIKU_ID = "anthropic.claude-3-haiku-20240307-v1:0"
HYPER_PARAMS = {"temperature": 0.35, "topP": .3}
REASONING_PATTERN = r'<considering>(.*?)</considering>'
CORRECTNESS_PATTERN = r'<reply>(.*?)</reply>'
def __init__(self):
self.bedrock = boto3.shopper('bedrock-runtime')
def classify_tickets(self, tickets: Listing[Dict[str, str]]) -> Listing[Dict[str, str]]:
prompts = [self._create_chat_payload(t) for t in tickets]
responses = self._call_threaded(prompts, self._call_bedrock)
formatted_responses = [self._format_results(r) for r in responses]
return [{**d1, **d2} for d1, d2 in zip(tickets, formatted_responses)]
def _call_bedrock(self, message_list: record[dict]) -> str:
response = self.bedrock.converse(
modelId=self.HAIKU_ID,
messages=message_list,
inferenceConfig=self.HYPER_PARAMS,
system=[{"text": SYSTEM_PROMPT}]
)
return response['output']['message']['content'][0]['text']
def _call_threaded(self, requests, perform):
future_to_position = {}
with ThreadPoolExecutor(max_workers=5) as executor:
for i, request in enumerate(requests):
future = executor.submit(perform, request)
future_to_position[future] = i
responses = [None] * len(requests)
for future in as_completed(future_to_position):
place = future_to_position[future]
strive:
response = future.outcome()
responses[position] = response
besides Exception as exc:
print(f"Request at place {place} generated an exception: {exc}")
responses[position] = None
return responses
def _create_chat_payload(self, ticket: dict) -> dict:
user_prompt = USER_PROMPT.format(abstract=ticket['Summary'], description=ticket['Description'])
user_msg = {"function": "person", "content material": [{"text": user_prompt}]}
return [user_msg]
def _format_results(self, model_response: str) -> dict:
reasoning = self._extract_with_regex(model_response, self.REASONING_PATTERN)
correctness = self._extract_with_regex(model_response, self.CORRECTNESS_PATTERN)
return {'Mannequin Reply': correctness, 'Reasoning': reasoning}
@staticmethod
def _extract_with_regex(response, regex):
matches = re.search(regex, response, re.DOTALL)
return matches.group(1).strip() if matches else None

Lastly, the labeled tickets are transformed to a CSV and uploaded to S3

import boto3
import io
import csv
s3 = boto3.shopper('s3')
def upload_csv(knowledge: Listing[Dict[str, str]]) -> None:
csv_buffer = io.StringIO()
author = csv.DictWriter(csv_buffer, fieldnames=knowledge[0].keys())
author.writeheader()
author.writerows(knowledge)
current_time = datetime.now().strftime("%Ypercentmpercentd_percentHpercentMpercentS")
filename = f"processed/processed_{current_time}.csv"
s3.put_object(
Bucket=self.bucket_name,
Key=filename,
Physique=csv_buffer.getvalue()
)

The mission is dashboard agnostic. Any fashionable software/service will work so long as it may possibly eat a CSV. Amazon Quicksight, Tableu or something in between will do.

On this weblog we mentioned utilizing Bedrock to routinely classify Jira tickets. These enriched tickets can then be used to create dashboards utilizing varied AWS Companies or 3P instruments. The takeaway, is that classifying textual content has change into a lot easier because the adoption of LLMs and what would have taken weeks can now be accomplished in days.

When you loved this text be happy to attach with me on LinkedIn