months on a Machine Studying mission, solely to find you by no means outlined the “right” downside initially? If that’s the case, or even when not, and you might be solely beginning with the info science or AI discipline, welcome to my first Ivory Tower Observe, the place I’ll deal with this subject.
The time period “Ivory Tower” is a metaphor for a state of affairs during which somebody is remoted from the sensible realities of on a regular basis life. In academia, the time period usually refers to researchers who have interaction deeply in theoretical pursuits and stay distant from the realities that practitioners face outdoors academia.
As a former researcher, I wrote a quick sequence of posts from my outdated Ivory Tower notes — the notes earlier than the LLM period.
Scary, I do know. I’m scripting this to handle expectations and the query, “Why ever did you do issues this fashion?” — “As a result of no LLM instructed me how you can do in any other case 10+ years in the past.”
That’s why my notes include “legacy” subjects resembling information mining, machine studying, multi-criteria decision-making, and (generally) human interactions, airplanes ✈️ and artwork.
Nonetheless, every time there is a chance, I’ll map my “outdated” information to generative AI advances and clarify how I utilized it to datasets past the Ivory Tower.
Welcome to submit #1…
How each Machine Studying and AI journey begins
— It begins with an issue.
For you, that is often “the” downside as a result of you want to dwell with it for months or, within the case of analysis, years.
With “the” downside, I’m addressing the enterprise downside you don’t absolutely perceive or know how you can resolve at first.
An excellent worse situation is while you assume you absolutely perceive and know how you can resolve it rapidly. This then creates solely extra issues which are once more solely yours to resolve. However extra about this within the upcoming sections.
So, what’s “the” downside about?
Causa: It’s largely about not managing or leveraging sources correctly — workforce, gear, cash, or time.
Ratio: It’s often about producing enterprise worth, which may span from improved accuracy, elevated productiveness, price financial savings, income positive aspects, quicker response, resolution, planning, supply or turnaround instances.
Veritas: It’s at all times about discovering an answer that depends and is hidden someplace within the current dataset.
Or, multiple dataset that somebody labelled as “the one”, and that’s ready so that you can resolve the downside. As a result of datasets comply with and are created from technical or enterprise course of logs, “there must be an answer mendacity someplace inside them.”
Ah, if solely it have been really easy.
Avoiding a unique chain of thought once more, the purpose is you will want to:
1 — Perceive the issue absolutely,
2 — If not given, discover the dataset “behind” it, and
3 — Create a strategy to get to the answer that can generate enterprise worth from it.
On this path, you may be tracked and measured, and time won’t be in your facet to ship the answer that can resolve “the universe equation.”
That’s why you will want to strategy the issue methodologically, drill all the way down to smaller issues first, and focus totally on them as a result of they’re the foundation reason for the general downside.
That’s why it’s good to discover ways to…
Assume like a Knowledge Scientist.
Returning to the issue itself, let’s think about that you’re a vacationer misplaced someplace within the huge museum, and also you need to determine the place you might be. What you do subsequent is stroll to the closest data map on the ground, which can present your present location.
At this second, in entrance of you, you see one thing like this:
The following factor you would possibly inform your self is, “I need to get to Frida Kahlo’s portray.” (Observe: These are the insights you need to get.)
As a result of your objective is to see this one portray that introduced you miles away from your private home and now sits two flooring beneath, you head straight to the second flooring. Beforehand, you memorized the shortest path to succeed in your objective. (Observe: That is the preliminary information assortment and discovery section.)
Nevertheless, alongside the way in which, you bump into some obstacles — the elevator is shut down for renovation, so you need to use the steps. The museum work have been reordered simply two days in the past, and the information plans didn’t replicate the modifications, so the trail you had in thoughts to get to the portray just isn’t correct.
Then you end up wandering across the third flooring already, asking quietly once more, “How do I get out of this labyrinth and get to my portray quicker?”
When you don’t know the reply, you ask the museum employees on the third flooring that can assist you out, and also you begin accumulating the brand new information to get the proper path to your portray. (Observe: This can be a new information assortment and discovery section.)
Nonetheless, when you get to the second flooring, you get misplaced once more, however what you do subsequent is begin noticing a sample in how the work have been ordered chronologically and thematically to group the artists whose types overlap, thus supplying you with a sign of the place to go to seek out your portray. (Observe: This can be a modelling section overlapped with the enrichment section from the dataset you collected throughout faculty days — your artwork information.)
Lastly, after adapting the sample evaluation and recalling the collected inputs on the museum route, you arrive in entrance of the portray you had been planning to see since reserving your flight a number of months in the past.
What I described now could be the way you strategy information science and, these days, generative AI issues. You at all times begin with the top objective in thoughts and ask your self:
“What’s the anticipated final result I need or have to get from this?”
Then you definitely begin planning from this query backwards. The instance above began with requesting holidays, reserving flights, arranging lodging, touring to a vacation spot, shopping for museum tickets, wandering round in a museum, after which seeing the portray you’ve been studying about for ages.
In fact, there may be extra to it, and this course of ought to be approached in another way if you want to resolve another person’s downside, which is a little more complicated than finding the portray within the museum.
On this case, you need to…
Ask the “good” questions.
To do that, let’s outline what a good query means [1]:
A good information science query have to be concrete, tractable, and answerable. Your query works nicely if it naturally factors to a possible strategy on your mission. In case your query is too imprecise to recommend what information you want, it received’t successfully information your work.
Formulating good questions retains you on observe so that you don’t get misplaced within the information that ought to be used to get to the precise downside answer, otherwise you don’t find yourself fixing the fallacious downside.
Going into extra element, good questions will assist determine gaps in reasoning, keep away from defective premises, and create various situations in case issues do go south (which nearly at all times occurs)👇🏼.
From the above-presented diagram, you perceive how good questions, at the start, have to assist concrete assumptions. This implies they must be formulated in a method that your premises are clear and guarantee they are often examined with out mixing up info with opinions.
Good questions produce solutions that transfer you nearer to your objective, whether or not by means of confirming hypotheses, offering new insights, or eliminating fallacious paths. They’re measurable, and with this, they connect with mission objectives as a result of they’re formulated with consideration of what’s potential, useful, and environment friendly [2].
Good questions are answerable with out there information, contemplating present information relevance and limitations.
Final however not least, good questions anticipate obstacles. If one thing is definite in information science, that is the uncertainty, so having backup plans when issues don’t work as anticipated is vital to supply outcomes on your mission.
Let’s exemplify this with one use case of an airline firm that has a problem with rising its fleet availability attributable to unplanned technical groundings (UTG).
These surprising upkeep occasions disrupt flights and value the corporate important cash. Due to this, executives determined to react to the issue and name in an information scientist (you) to assist them enhance plane availability.
Now, if this may be the primary information science job you ever obtained, you’d possibly begin an investigation by asking:
“How can we eradicate all unplanned upkeep occasions?”
You perceive how this query is an instance of the fallacious or “poor” one as a result of:
- It’s not real looking: It contains each potential defect, each small and large, into one not possible objective of “zero operational interruptions”.
- It doesn’t maintain a measure of success: There’s no concrete metric to indicate progress, and if you happen to’re not at zero, you’re at “failure.”
- It’s not data-driven: The query didn’t cowl which information is recorded earlier than delays happen, and the way the plane unavailability is measured and reported from it.
So, as a substitute of this imprecise query, you’d most likely ask a set of focused questions:
- Which plane (sub)system is most crucial to flight disruptions?
(Concrete, particular, answerable) This query narrows down your scope, specializing in just one or two particular (sub) methods affecting most delays. - What constitutes “crucial downtime” from an operational perspective?
(Priceless, ties to enterprise objectives) If the airline (or regulatory physique) doesn’t outline what number of minutes of unscheduled downtime matter for schedule disruptions, you would possibly waste effort fixing much less pressing points. - Which information sources seize the foundation causes, and the way can we fuse them?
(Manageable, narrows the scope of the mission additional) This clarifies which information sources one would want to seek out the issue answer.
With these sharper questions, you’ll drill all the way down to the true downside:
- Not all delays weigh the identical in price or influence. The “right” information science downside is to foretell crucial subsystem failures that result in operationally pricey interruptions so upkeep crews can prioritize them.
That’s why…
Defining the issue determines each step after.
It’s the muse upon which your information, modelling, and analysis phases are constructed 👇🏼.
It means you might be clarifying the mission’s targets, constraints, and scope; you want to articulate the final word objective first and, apart from asking “What’s the anticipated final result I need or have to get from this?”, ask as nicely:
What would success seem like and the way can we measure it?
From there, drill all the way down to (potential) next-level questions that you just (I) have discovered from the Ivory Tower days:
— Historical past questions: “Has anybody tried to resolve this earlier than? What occurred? What continues to be lacking?”
— Context questions: “Who’s affected by this downside and the way? How are they partially resolving it now? Which sources, strategies, and instruments are they utilizing now, and may they nonetheless be reused within the new fashions?”
— Influence Questions: “What occurs if we don’t resolve this? What modifications if we do? Is there a price we will create by default? How a lot will this strategy price?”
— Assumption Questions: “What are we taking without any consideration which may not be true (particularly in relation to information and stakeholders’ concepts)?”
— ….
Then, do that within the loop and at all times “ask, ask once more, and don’t cease asking” questions so you possibly can drill down and perceive which information and evaluation are wanted and what the bottom downside is.
That is the evergreen information you possibly can apply these days, too, when deciding in case your downside is of a predictive or generative nature.
(Extra about this in another be aware the place I’ll clarify how problematic it’s making an attempt to resolve the issue with the fashions which have by no means seen — or have by no means been educated on — comparable issues earlier than.)
Now, going again to reminiscence lane…
I need to add one vital be aware: I’ve discovered from late nights within the Ivory Tower that no quantity of information or information science information can prevent if you happen to’re fixing the fallacious downside and making an attempt to get the answer (reply) from a query that was merely fallacious and imprecise.
When you’ve an issue available, don’t rush into assumptions or constructing the fashions with out understanding what you want to do (Festina lente).
As well as, put together your self for surprising conditions and do a correct investigation together with your stakeholders and area specialists as a result of their persistence might be restricted, too.
With this, I need to say that the “actual artwork” of being profitable in information initiatives is understanding exactly what the issue is, determining if it may be solved within the first place, after which arising with the “how” half.
You get there by studying to ask good questions.
If I got one hour to avoid wasting the planet, I’d spend 59 minutes defining the issue and one minute fixing it.
Thanks for studying, and keep tuned for the subsequent Ivory Tower be aware.
For those who discovered this submit useful, be at liberty to share it together with your community. 👏
Join for extra tales on Medium ✍️ and LinkedIn 🖇️.
References:
[1] DS4Humans, Backwards Design, accessed: April fifth 2025, https://ds4humans.com/40_in_practice/05_backwards_design.html#defining-a-good-question
[2] Godsey, B. (2017), Assume Like a Knowledge Scientist: Sort out the info science course of step-by-step, Manning Publications.