Data Engineering

One thing I learned from my work is that even though having proper table definition is good for the data standardization, sometimes we just want to have the data be ingested to Snowflake without worrying about if all the column types are 100% correct or not. The following is the code I use for this case which can help me easily ingest data into snowflake with only 4 main required parameters, SOURCE_NAME, TARGET_NAME, FILE_FORMAT, and ALL_COLUMN_TYPE_AS_STRING.

Few days ago, I asked myself a question about how can you better manage all the database changes defined in SQL scripts with DDL (Data Definition Language)? Maybe try to use version control on the SQL code and then use DevOps pipelines to achieve continuous deployment. Then, I came up with a poc solution, choux130/DevOps_In_DE/jenkins_mysql_flyway. This is the architecture overview, Here are some undo works on this poc and I wish I can improve it in the future.

In 2017, I started my first job as an Engineer Intern in the Predictive Analytics team, and after a year, I was promoted to Data Engineer. But to be honest, I was never sure if the titles I got can really describe my role in ChannelAdvisor since I was the first Data Engineer in our company. There were no clearly defined responsibilities in this role at the beginning and the responsibilities changes based on the tasks I did.

It is time to practice T-SQL but how can I do that with my Macbook Pro? Thanks to Docker, it is never mission impossible. References Running SQL Server 2019 CTP in a Docker container – DBA From The Cold Running SQL Server with Docker on the Mac – SQL passion Steps Follow the post to install docker and create a container with SQL Server 2019 CTP. Install Azure Data Studio and read the post Download Wide World Importers sample database (WideWorldImporters-Full.

When I am working on my personal projects, all the data I have can be easily saved as .csv or .txt files. Things are not complicated. However, when things are in scale, it is hard to not to talk about database. In my company, all the data are stored in SQL Server database. So, it is important for me to know how to connect R with SQL Server. And, also good to know how to interact with database in R by passing variables in R environment to database and then return the desired dataset.

This page is about the SQL Server commands. All of the contents are based on the code in w3schools - SQL with the sample database, Northwind. To make myself easier to look up the commands I want in the future, I picked out the ones I have been used a lot in my work and add some notes on it. I also borrowed some materials from the course LinkdedIn Learning - Microsoft SQL Server 2016: Query Data.