Welcome to my blog, where I write about the design and problem-solving processes behind modern data engineering solutions.
Recent Data Engineering
A Comically Complicated Adventure in Setting up a Cloud SFTP Server
Our team was recently presented with a project that required some infrastructure work–in particlar, one step in the project required a data transfer that needed to happen via sftp. As we didn’t currently have an sftp server, we needed to get one set up.
In theory this seemed like an easy problem to solve, but it turned into quite a saga. Part of the difficulties were due to my own misunderstanding of the infrastructure on the side of the data owner; I made assumptions without fully fleshing out the details beforehand.
read more