Cloud

In my daily role I am a software engineer at Union focusing on backend development. We are the core maintainers, and the most frequent contributors, to the open-source Flyte project. Per our documentation it is described as “The Workflow Automation Platform for Complex, Mission-Critical Data and Machine Learning Processes at Scale”. Basically, it is a framework to abstract execution of complex data workflows and the coinciding cloud infrastructure management. This enables teams to efficiently and effectively scale data processing....

I have fallen a bit behind on my 100 days to offload pace. Certainly not from lack of interesting discoveries or happenings, rather too many overlapping my time to write. I’m hoping to get back on track with this post about Delta Lake, databricks open-sourced data lakehouse implementation. This discussion surrounds the paper Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores. I highly recommend the read, in doing so I gained deeper insight into the direction of data storage including specific industry use-cases and the relationships between storage solutions thereof....