Seize context and enhance predictions with historic information
I not too long ago made a profession shift barely away from information science to a extra engineering-heavy position. My workforce is constructing a high-quality information warehouse to feed the group’s BI and ML wants. I took this place as a result of I noticed it as a possibility to make use of the perception I’ve gained from my information science roles to affect the design and improvement of a knowledge warehouse with a forward-looking focus.
In almost each information science position I’ve labored in over the previous 6 years, I seen a typical theme — the information infrastructure wasn’t designed with information science in thoughts. Many tables within the information warehouse/lakehouse, normally information and dimensions, lacked crucial fields or construction required to construct performant machine studying fashions. Essentially the most prevalent limitation I seen was that almost all tables solely tracked the present state of an statement slightly than the historical past.
This text explores this frequent drawback and demonstrates the way to tackle it via a knowledge modeling approach referred to as Slowly Altering Dimensions. By the top, you’ll perceive the impression of storing historic information in your mannequin’s efficiency, and also you’ll have methods that will help you implement this in your use circumstances.