The primary workflow is made up of two nested sub-workflows:
summary_gen: This sub-workflow finds analysis papers on the given subject and generates summaries. It carries out the trying to find papers by net querying and makes use of LLM to get insights and summaries as instructed.slide_gen: this sub-workflow is chargeable for producing a PowerPoint slide deck utilizing the summaries from the earlier step. It codecs the slides utilizing a offered PowerPoint template and generates them by creating and executing Python code utilizing thepython-pptxlibrary.
Let’s take a more in-depth have a look at the these sub-workflows. Firstly, the summary_gen workflow, which is fairly simple. It follows a easy linear course of. It principally serves as a “knowledge processing” workflow, with some steps sending a request to an LLM.
The workflow begins by getting a consumer enter (a analysis subject) and run by means of the next steps:
tavily_query: Queries with the Tavily API to get educational papers associated to the subject as a structured response.get_paper_with_citations: For every paper returned from the Tavily question, the step retrieves the paper metadata together with that of the cited paper utilizing the SemanticScholar API.filter_papers: Since not all citations retrieved are straight related to the unique subject, this step refines the consequence. The titles and abstracts of every paper are despatched to the LLM to evaluate their relevance. This step is outlined as:
@step(num_workers=4)
async def filter_papers(self, ev: PaperEvent) -> FilteredPaperEvent:
llm = new_gpt4o_mini(temperature=0.0)
response = await process_citation(ev.paper, llm)
return FilteredPaperEvent(paper=ev.paper, is_relevant=response)
Right here within the process_citation() operate, we use the FunctionCallingProgram from LlamaIndex to get a structured response:
IS_CITATION_RELEVANT_PMT = """
You assist a researcher determine whether or not a paper is related to their present analysis subject: {subject}
You're given the title and summary of a paper.
title: {title}
summary: {summary}Give a rating indicating the relevancy to the analysis subject, the place:
Rating 0: Not related
Rating 1: Considerably related
Rating 2: Very related
Reply with integer rating 0, 1 or 2 and your cause.
"""
class IsCitationRelevant(BaseModel):
rating: int
cause: str
async def process_citation(quotation, llm):
program = FunctionCallingProgram.from_defaults(
llm=llm,
output_cls=IsCitationRelevant,
prompt_template_str=IS_CITATION_RELEVANT_PMT,
verbose=True,
)
response = await program.acall(
title=quotation.title,
summary=quotation.abstract,
subject=quotation.subject,
description="Information mannequin for whether or not the paper is related to the analysis subject.",
)
return response
download_papers: This step gathers all filtered papers, prioritizes them primarily based on relevance rating and availability on ArXiv, and downloads probably the most related ones.paper2summary_dispatcher: Every downloaded paper is ready for abstract technology by establishing paths for storing the pictures and the summaries. This step makes use ofself.send_event()to allow the parallel execution of thepaper2summarystep for every paper. It additionally units the variety of papers within the workflow context with a variablectx.knowledge[“n_pdfs”]in order that the later steps know what number of papers they’re anticipated to course of in complete.
@step(pass_context=True)
async def paper2summary_dispatcher(
self, ctx: Context, ev: Paper2SummaryDispatcherEvent
) -> Paper2SummaryEvent:
ctx.knowledge["n_pdfs"] = 0
for pdf_name in Path(ev.papers_path).glob("*.pdf"):
img_output_dir = self.papers_images_path / pdf_name.stem
img_output_dir.mkdir(exist_ok=True, dad and mom=True)
summary_fpath = self.paper_summary_path / f"{pdf_name.stem}.md"
ctx.knowledge["n_pdfs"] += 1
self.send_event(
Paper2SummaryEvent(
pdf_path=pdf_name,
image_output_dir=img_output_dir,
summary_path=summary_fpath,
)
)
paper2summary: For every paper, it converts the PDF into photographs, that are then despatched to the LLM for summarization. As soon as the abstract is generated, it’s saved in a markdown file for future reference. Significantly, the abstract generated right here is kind of elaborated, like a small article, so not fairly appropriate but for placing straight within the presentation. However it’s stored in order that the consumer can view these intermediate outcomes. In one of many later steps, we are going to make this info extra presentable. The immediate offered to the LLM contains key directions to make sure correct and concise summaries:
SUMMARIZE_PAPER_PMT = """
You're an AI specialised in summarizing scientific papers.
Your objective is to create concise and informative summaries, with every part ideally round 100 phrases and
restricted to a most of 200 phrases, specializing in the core strategy, methodology, datasets,
analysis particulars, and conclusions offered within the paper. After you summarize the paper,
save the abstract as a markdown file.Directions:
- Key Method: Summarize the primary strategy or mannequin proposed by the authors.
Give attention to the core concept behind their technique, together with any novel strategies, algorithms, or frameworks launched.
- Key Parts/Steps: Establish and describe the important thing parts or steps within the mannequin or strategy.
Break down the structure, modules, or phases concerned, and clarify how every contributes to the general technique.
- Mannequin Coaching/Finetuning: Clarify how the authors skilled or finetuned their mannequin.
Embrace particulars on the coaching course of, loss capabilities, optimization strategies,
and any particular methods used to enhance the mannequin’s efficiency.
- Dataset Particulars: Present an outline of the datasets used within the research.
Embrace info on the dimensions, sort and supply. Point out whether or not the dataset is publicly obtainable
and if there are any benchmarks related to it.
- Analysis Strategies and Metrics: Element the analysis course of used to evaluate the mannequin's efficiency.
Embrace the strategies, benchmarks, and metrics employed.
- Conclusion: Summarize the conclusions drawn by the authors. Embrace the importance of the findings,
any potential purposes, limitations acknowledged by the authors, and prompt future work.
Be certain that the abstract is obvious and concise, avoiding pointless jargon or overly technical language.
Intention to be comprehensible to somebody with a common background within the subject.
Be certain that all particulars are correct and faithfully signify the content material of the unique paper.
Keep away from introducing any bias or interpretation past what's offered by the authors. Don't add any
info that's not explicitly said within the paper. Follow the content material offered by the authors.
"""
end: The workflow collects all generated summaries, verifies they’re accurately saved, and logs the completion of the method, and return aStopEventas a ultimate consequence.
If this workflow have been to run independently, execution would finish right here. Nonetheless, since that is only a sub-workflow of the primary course of, upon completion, the subsequent sub-workflow — slide_gen — is triggered.
This workflow generates slides primarily based on the summaries created within the earlier step. Right here is an outline of the slide_gen workflow:
When the earlier sub-workflow finishes, and the abstract markdown information are prepared, this workflow begins:
get_summaries: This step reads the content material of the abstract information, triggers aSummaryEventfor every file using once moreself.send_event()to allow concurrent execution for quicker processing.summary2outline: This step makes summaries into slide define texts by utilizing LLM. It shortens the summaries into sentences or bullet factors for placing within the presentation.gather_feedback_outline: On this step, it presents the the consumer the proposed slide define alongside the paper abstract for them to assessment. The consumer gives suggestions, which can set off anOutlineFeedbackEventif revisions are crucial. This suggestions loop continues with thesummary2outlinestep till the consumer approves the ultimate define, at which level anOutlineOkEventis triggered.
@step(pass_context=True)
async def gather_feedback_outline(
self, ctx: Context, ev: OutlineEvent
) -> OutlineFeedbackEvent | OutlineOkEvent:
"""Current consumer the unique paper abstract and the outlines generated, collect suggestions from consumer"""
print(f"the unique abstract is: {ev.abstract}")
print(f"the define is: {ev.define}")
print("Do you wish to proceed with this define? (sure/no):")
suggestions = enter()
if suggestions.decrease().strip() in ["yes", "y"]:
return OutlineOkEvent(abstract=ev.abstract, define=ev.define)
else:
print("Please present suggestions on the define:")
suggestions = enter()
return OutlineFeedbackEvent(
abstract=ev.abstract, define=ev.define, suggestions=suggestions
)
outlines_with_layout: It augments each slide define by together with web page format particulars from the given PowerPoint template, utilizing LLM. This stage saves the content material and design for all slide pages in a JSON file.slide_gen: It makes use of a ReAct agent to make slide decks primarily based on given outlines and format particulars. This agent has a code interpreter software to run and proper code in an remoted surroundings and a layout-checking software to take a look at the given PowerPoint template info. The agent is prompted to make use ofpython-pptxto create the slides and might observe and repair errors.
@step(pass_context=True)
async def slide_gen(
self, ctx: Context, ev: OutlinesWithLayoutEvent
) -> SlideGeneratedEvent:
agent = ReActAgent.from_tools(
instruments=self.azure_code_interpreter.to_tool_list() + [self.all_layout_tool],
llm=new_gpt4o(0.1),
verbose=True,
max_iterations=50,
)immediate = (
SLIDE_GEN_PMT.format(
json_file_path=ev.outlines_fpath.as_posix(),
template_fpath=self.slide_template_path,
final_slide_fname=self.final_slide_fname,
)
+ REACT_PROMPT_SUFFIX
)
agent.update_prompts({"agent_worker:system_prompt": PromptTemplate(immediate)})
res = self.azure_code_interpreter.upload_file(
local_file_path=self.slide_template_path
)
logging.information(f"Uploaded file to Azure: {res}")
response = agent.chat(
f"An instance of define merchandise in json is {ev.outline_example.json()},"
f" generate a slide deck"
)
local_files = self.download_all_files_from_session()
return SlideGeneratedEvent(
pptx_fpath=f"{self.workflow_artifacts_path}/{self.final_slide_fname}"
)
validate_slides: Checks the slide deck to verify it meets the given requirements. This step entails turning the slides into photographs and having the LLM visually examine them for proper content material and constant fashion in line with the rules. Relying on what the LLM finds, it would both ship out aSlideValidationEventif there are issues or aStopEventif all the things seems to be good.
@step(pass_context=True)
async def validate_slides(
self, ctx: Context, ev: SlideGeneratedEvent
) -> StopEvent | SlideValidationEvent:
"""Validate the generated slide deck"""
ctx.knowledge["n_retry"] += 1
ctx.knowledge["latest_pptx_file"] = Path(ev.pptx_fpath).identify
img_dir = pptx2images(Path(ev.pptx_fpath))
image_documents = SimpleDirectoryReader(img_dir).load_data()
llm = mm_gpt4o
program = MultiModalLLMCompletionProgram.from_defaults(
output_parser=PydanticOutputParser(SlideValidationResult),
image_documents=image_documents,
prompt_template_str=SLIDE_VALIDATION_PMT,
multi_modal_llm=llm,
verbose=True,
)
response = program()
if response.is_valid:
return StopEvent(
self.workflow_artifacts_path.joinpath(self.final_slide_fname)
)
else:
if ctx.knowledge["n_retry"] < self.max_validation_retries:
return SlideValidationEvent(consequence=response)
else:
return StopEvent(
f"The slides aren't mounted after {self.max_validation_retries} retries!"
)
The factors used for validation are:
SLIDE_VALIDATION_PMT = """
You're an AI that validates the slide deck generated in line with following guidelines:
- The slide must have a entrance web page
- The slide must have a ultimate web page (e.g. a 'thanks' or 'questions' web page)
- The slide texts are clearly readable, not reduce off, not overflowing the textbox
and never overlapping with different componentsIf any of the above guidelines are violated, it's essential to present the index of the slide that violates the rule,
in addition to suggestion on the best way to repair it.
"""
modify_slides: Ought to the slides fail the validation test, the earlier step sends aSlideValidationEventoccasion. Right here one other ReAct agent updates the slides in line with validator suggestions, with the up to date slides being saved and returned to be validated once more. This verification loop might happen a number of occasions in line with themax_validation_retriesvariable attributes of theSlideGenWorkflowclass.