Why would anyone want to become a Data Engineer in 2020?
Daniel Collett , Founder of Integ Professional asked two highly experienced Data Engineers Ex Deliveroo and Expedia various questions on the 2020 Data Engineering landscape.
Candidate 1
Why would anyone want to become a Data Engineer in 2020?
If you have an appetite for data and a passion for building data systems that increases your company’s annual profits by several millions a year, then you may want to consider a career path in Data Engineering. Likewise if you enjoy extracting and analysing data to create data solutions, then this may be the job for you. Since 2018 to the present date, there has been an increasing demand for Data Engineers. Intermediate level Data Engineers can earn up to 80k and highly experienced Data Engineers can earn in excess of £150k. This income is attractive to some and therefore the demand to become a Data Engineer in 2020 is certainly on the rise.
What are the responsibilities of Data Engineers?
Data Engineers set up and maintain the data infrastructures that support business information systems and applications. This can vary from being employed by an SME where they use a small database, or for a larger corporate company where they use a larger database i.e Petabyte-scale data warehouse.
Data Engineers have the responsibility to design, build and implement the data systems that fuel machine learning and AI analytics. They develop information processes for data extraction, data transformation, data migration, data verification, data modelling and data mining. Their programming skills can vary from using Java, Python and R. They are normally highly skilled in SQL and NoSQL database systems and use distributed systems such as Hadoop.
Ascertaining a mixture of skills allows them to work with the database administrators, architects and data scientists to build a robust, integrated data infrastructure for any business.
How Data Engineers Support Data Scientists:
Data Engineers create data pipelines that connect data from one system to another and they transform data from one format to another. That way, different data systems can communicate so the data scientist can pull the analytical insights s/he desires. Even though Data Engineers aren’t as visible as data scientists, they’re just as important (if not more so).
What are the top traits you believe makes up an experienced Data Engineer? (Opinions below)
- You must get into the habit of being strong at relationship building and communication within a company, not just technical, but be able to communicate verbally with different people of all levels and keeping those relationships strong. Throughout most of my career I needed to work closely with data scientists, DBA’s, administrators, software engineers and had to be able to communicate with them all.
- Expert in SQL language is a must – It is the universal language of data.
- An expert in a Programming language, usually Python, is required. At any level – The basics of python is a must.
- Strong at data mining and a Strong Data sense, as you must be able to make sense what data means and therefore you need an analytical mind similar to data analysts.
- A varied Technology landscape; need to understand trends in the data world and keep up to date with what’s new/what tools and methodologies are emerging/consolidating. Companies differ and have different needs.
- Able to be a mentor and deal with coaching/driving colleagues, communicating and transferring knowledge; as mentioned before it would be good to be an expert in at least one programming language.
- Education is important and often companies like a formal degree, however a lack of it may be replaced by solid working experience.
- Constantly learning – Be active in Forums and attend events to understand what other people/companies are doing.
Candidate 2
Why would anyone want to become a Data Engineer in 2020?
Data Engineering gives you the opportunity to work with lots of new and exciting technologies. The field is still quite young and as a Data engineer you get the chance to implement new technologies and help discover best practices for working with them, and having the tools to properly collect and interpret your data is key. Many companies realise that Data engineers are in high demand so they make recruiting talented Data Engineers a high priority.
Your work is key in supporting data driven decision making.
What are the responsibilities of a Data Engineer?
The role of a Data Engineer can be varied and includes; connecting to new data sources, maintaining and streaming the platform, building streaming applications, loading data into the data warehouse, transforming data once its in the Data warehouse, building reports and everything in between. Most Data Engineers will not do all of the above, but will lean more towards maintaining the platform or maintaining ETL logic and reports in the Data warehouse.
Regardless of their exact responsibilities, a Data Engineer should be able to leverage modern big data technologies, which can scale to the ever increasing quantities of data being processed in a day.
What are the top traits you believe makes up an experienced Data Engineer? (Opinions below)
- Thirst for knowledge – There is always lots to learn!
- Attention to detail – Good for any role, but especially useful for a Data Engineer.
- Persistence – You will encounter some pretty stubborn bugs.
- Willing to help – As a Data Engineer people will ask you questions, a lot!
- Relentless high standards – Building accurate and reliable data pipelines is hard.
- Good communicator – You have the best understanding of the data and can communicate effectively.
If you would like to be featured on our blog, interlinked and published on our Linkedin which has a large network, please GET IN TOUCH
