Learn to use Google’s Germini-1.5-pro-latest mannequin to develop a generative AI app for calorie counting
Have you ever ever questioned the quantity of energy you eat whenever you eat your dinner, for instance? I do this on a regular basis. Wouldn’t or not it’s fantastic in the event you might merely cross an image of your plate by way of an app and get an estimate of the whole variety of energy earlier than you resolve how far in you need to dip?
This calorie counter app that I created will help you obtain this. It’s a Python software that makes use of Google’s Gemini-1.5-Professional-Newest mannequin to estimate the variety of energy in meals gadgets.
The app takes two inputs: a query concerning the meals and a picture of the meals or meals gadgets, or just, a plate of meals. It outputs a solution to the query, the whole variety of energy within the picture and a breakdown of energy by every meals merchandise within the picture.
On this article, I’ll clarify the complete end-to-end technique of constructing the app from scratch, utilizing Google’s Gemini-1.5-pro-latest (a Giant Language generative AI mannequin launched by Google), and the way I developed the front-end of the appliance utilizing Streamlit.
It’s value noting right here that with developments on the earth of AI, it’s incumbent on knowledge scientists to steadily shift from conventional deep studying to generative AI strategies so as to revolutionize their function. That is my major objective of teaching on this topic.
Let me begin by briefly explaining Gemini-1.5-pro-latest and the streamlit framework, as they’re the most important parts within the infrastructure of this calorie counter app.
Gemini-1.5-pro-latest is a sophisticated AI language mannequin developed by Google. Since it’s the newest model, it has enhanced capabilities over earlier variations within the mild of quicker response instances and improved accuracy when utilized in pure language processing and constructing functions.
It is a multi-modal mannequin that works with each texts and pictures — an development from Google Gemini-pro mannequin which solely works with textual content prompts.
The mannequin works by understanding and producing textual content, like people, primarily based on prompts given to it. On this article, this mannequin will likely be used to to generate textual content for our energy counter app.
Gemini-1.5-pro-latest could be built-in into different functions to strengthen their AI capabilities. On this present software, the mannequin makes use of generative AI strategies to interrupt the uploaded picture into particular person meals gadgets . Based mostly on its contextual understanding of the meals gadgets from its dietary database, it makes use of picture recognition and object detection to estimate the variety of energy, after which totals up the energy for all gadgets within the picture.
Streamlit is an open-source Python framework that may handle the person interface. This framework simplifies internet growth in order that all through the challenge, you do not want to put in writing any HTML and CSS codes for the entrance finish.
Allow us to dive into constructing the app.
I’ll present you construct the app in 5 clear steps.
1. Arrange your Folder construction
For a begin, go into your favourite code editor (mine is VS Code) and begin a challenge file. Name it Energy-Counter, for instance. That is the present working listing. Create a digital surroundings (venv), activate it in your terminal, after which create the next information: .env, energy.py, necessities.txt.
Right here’s a advice for the look of your folder construction:
Energy-Counter/
├── venv/
│ ├── xxx
│ ├── xxx
├── .env
├── energy.py
└── necessities.txt
Please observe that Gemini-1.5-Professional works finest with Python variations 3.9 and better.
2. Get the Google API key
Like different Gemini fashions, Gemini-1.5-pro-latest is at present free for public use. Accessing it requires that you just receive an API key, which you may get from Google AI Studio by going to “Get API key” on this hyperlink. As soon as the bottom line is generated, copy it for subsequent use in your code. Save this key as an surroundings variable within the .env file as follows.
GOOGLE_API_KEY="paste the generated key right here"
3. Set up dependencies
Kind the next libraries into your necessities.txt file.
- streamlit
- google-generativeai
- python-dotenv
Within the terminal, set up the libraries in necessities.txt with:
python -m pip set up -r necessities.txt
4. Write the Python script
Now, let’s begin writing the Python script in energy.py. With the next code, import all required libraries:
# import the libraries
from dotenv import load_dotenv
import streamlit as st
import os
import google.generativeai as genai
from PIL import Picture
Right here’s how the varied modules imported will likely be used:
- dotenv — Since this software will likely be configured from a Google API key surroundings variable, dotenv is used to load configuration from the .env file.
- Streamlit — to create an interactive person interface for front-end
- os module is used to deal with the present working listing whereas performing file operations like getting the API key from the .env file
- google.generativeai module, after all, provides us entry to the Gemini mannequin we’re about to make use of.
- PIL is a Python imaging library used for managing picture file codecs.
The next traces will configure the API keys and cargo them from the surroundings variables retailer.
genai.configure(api_key=os.getenv("GOOGLE_API_KEY"))load_dotenv()
Outline a operate that, when known as, will load the Gemini-1.5-pro-latest and get the response, as follows:
def get_gemini_reponse(input_prompt,picture,user_prompt):
mannequin=genai.GenerativeModel('gemini-1.5-pro-latest')
response=mannequin.generate_content([input_prompt,image[0],user_prompt])
return response.textual content
Within the above operate, you see that it takes as enter, the enter immediate that will likely be specified additional down within the script, a picture that will likely be equipped by the person, and a person immediate/query that will likely be equipped by the person. All that goes into the gemini mannequin to return the response textual content.
Since Gemini-1.5-pro expects enter pictures within the type of byte arrays, the following factor to do is write a operate that processes the uploaded picture, changing it to bytes.
def input_image_setup(uploaded_file):
# Examine if a file has been uploaded
if uploaded_file isn't None:
# Learn the file into bytes
bytes_data = uploaded_file.getvalue()image_parts = [
{
"mime_type": uploaded_file.type, # Get the mime type of the uploaded file
"data": bytes_data
}
]
return image_parts
else:
increase FileNotFoundError("No file uploaded")
Subsequent, specify the enter immediate that may decide the behaviour of your app. Right here, we’re merely telling Gemini what to do with the textual content and picture that the app will likely be fed with by the person.
input_prompt="""
You're an skilled nutritionist.
It's best to reply the query entered by the person within the enter primarily based on the uploaded picture you see.
You must also take a look at the meals gadgets discovered within the uploaded picture and calculate the whole energy.
Additionally, present the small print of each meals merchandise with energy consumption within the format beneath:1. Merchandise 1 - no of energy
2. Merchandise 2 - no of energy
----
----
"""
The following step is to initialize streamlit and create a easy person interface in your calorie counter app.
st.set_page_config(page_title="Gemini Calorie Counter App")
st.header("Calorie Counter App")
enter=st.text_input("Ask any query associated to your meals: ",key="enter")
uploaded_file = st.file_uploader("Add a picture of your meals", kind=["jpg", "jpeg", "png"])
picture=""
if uploaded_file isn't None:
picture = Picture.open(uploaded_file)
st.picture(picture, caption="Uploaded Picture.", use_column_width=True) #present the picturesubmit=st.button("Submit & Course of") #creates a "Submit & Course of" button
The above steps have all of the items of the app. At this level, the person is ready to open the app, enter a query and add a picture.
Lastly, let’s put all of the items collectively such that after the “Submit & Course of” button is clicked, the person will get the required response textual content.
# As soon as submit&Course of button is clicked
if submit:
image_data=input_image_setup(uploaded_file)
response=get_gemini_reponse(input_prompt,image_data,enter)
st.subheader("The Response is")
st.write(response)
5. Run the script and work together together with your app
Now that the app growth is full, you may execute it within the terminal utilizing the command:
streamlit run energy.py
To work together together with your app and see the way it performs, view your Streamlit app in your browser utilizing the native url or community URL generated.
This how your Streamlit app seems to be like when it’s first opened on the browser.
As soon as the person asks a query and uploads a picture, right here is the show:
As soon as the person pushes the “Submit & Course of” button, the response within the picture beneath is generated on the backside of the display.
For exterior entry, take into account deploying your app utilizing cloud companies like AWS, Heroku, Streamlit Group Cloud. On this case, let’s use Streamlit Group Cloud to deploy the app totally free.
On the highest proper of the app display, click on ‘Deploy’ and observe the prompts to finish the deployment.
After deployment, you may share the generated app URL to different customers.
Similar to different AI functions, the outcomes outputed are one of the best estimates of the mannequin, so, earlier than utterly counting on the app, please observe the next as a few of the potential dangers:
- The calorie counter app might misclassify sure meals gadgets and thus, give the improper variety of energy.
- The app doesn’t have a reference level to estimate the dimensions of the meals — portion — primarily based on the uploaded picture. This will result in errors.
- Over-reliance on the app can result in stress and psychological well being points as one might turn out to be obsessive about counting energy and worrying about outcomes that is probably not too correct.
To assist cut back the dangers that include utilizing the calorie counter, listed here are potential enhancements that might be built-in into its growth:
- Including contextual evaluation of the picture, which can assist to gauge the dimensions of the meals portion being analysed. As an example, the app might be constructed such that a normal object like a spoon, included within the meals picture, might be used as a reference level for measuring the sizes of the meals gadgets. It will cut back errors in ensuing complete energy.
- Google might enhance the variety of the meals gadgets of their coaching set to cut back misclassification errors. They may develop it to incorporate meals from extra cultures in order that even uncommon African meals gadgets will likely be recognized.