Possibilities of how Cloud solutions can impact IT Management & Governance

Why IT managers shouldn’t be threatened when seeing clouds on the horizon — Photo by Caryle Barton on Unsplash

Improve your IT Assets, Resources and Capabilities to enhance your Business Success. The management of a company must be sure that their IT adequately supports the company’s goals. IT Management and a good IT Governance is responsible for this. The Cloud and its’ commoditization of IT assets and resources have massive impacts on the whole IT Governance. Especially smaller businesses and Start-ups can profit. How the cloud can help to meet the company’s goals, will be answered in the following article [1].

The figure below will show a short and superficially overview of how IT Governance is defined [2][3][4]:


What are the Dependencies to the Source Systems?

Photo by John Fowler on Unsplash

When integrating data from system A to system B, data engineers and other stakeholders should not only focus on the data process, e.g. via ETL/ELT, but also on the source system. What various circumstances must be taken into account and what I learned from earlier projects are the following:

When is a source system available? You have to consider maintenance cycles, downtimes, etc. Otherwise, if the system is not available, the data integration process will not work or only part of the data will be captured. Here, it makes sense to implement a monitoring of the source system and work…


How to realize Big Data Projects

Photo by Hannah Vorenkamp on Unsplash

When setting up a Big Data landscape, there are five steps and topic blocks that must be taken into account during implementation.

In order to process data in a data lake or data warehouse, to analyze it or to make it usable for other systems, data must first be made available from the source systems. Examples for sources could be:

  • Internal systems like SAP, Salesforce, etc.
  • Internal databases from Oracle, Microsoft, MySQL etc.
  • Archived Files and Log Files
  • Documents
  • Social Media (like Facebook or Instagram API)
  • Web scraping data
  • Open API/Data

Beside classical batch and ETL process data integration (e.g…


What are the common Issues and how can they be solved?

Photo by Taylor Friehl on Unsplash

In the field of Data Analytics and related topics like BI, Data Science, Data Engineering etc. you often will hear about the same problems when working in a project or on a product. Here, I want to share my experiences and possible solutions.

One of the most unpleasant moments in the life of every project or product manager is when the business department complains about the data quality. The problems can be of different nature. Errors in the source system, ETL process or in the report.

Solution: Here, it is a good idea to set up a monitoring system and…


How to Anonymize and Pseudonymize Data

Photo by Francisco Suarez on Unsplash

Personal data is the core concept of data protection. Data protection law only applies when data relates to individuals. The GDPR for example increases fines to up to 20 million euros or, in the case of large companies and groups, up to 4% of the global group turnover of the previous year [1]. When working in the field of Big Data, Data Science or related fields it is essential to know about these laws and how anonymization and pseudonymization give the possibility of still using the data for your use cases.

This is any information relating to an identified or…


How to gain Insights on new Visualizing Techniques

Photo by Nikolay Maslov on Unsplash

In the world of Big Data, data visualization tools and techniques are essential to analyze large amounts of information and make data-driven decisions as data is increasingly used for important management decisions. So there is a trend away from gut feeling and emotional decisions towards rational choices that are made based on numbers. Therefore, reports and visualizations have to be easily understood and meaningful.

It is increasingly beneficial for professionals to be able to use data to make decisions and visuals to tell stories that communicate how data informs the question of person, subject, time, place, and method [1]. In…


Why FTP is still alive and how you can implement it into a modern Cloud Data Platform

Photo by Egor Myznik on Unsplash

The File Transfer Protocol is for the communication of people and devices over the Internet and other networks works through protocols [1]. Because FTP is an older method of data transfer, such transfers are compatible with many legacy and/or on-premises HR and business systems, making it a useful option if you want to integrate an older system with newer, cloud-based software.

In the past, most digital systems were connected via FTP integration, where one system exports data in a “flat file” format (often a spreadsheet) and another system imports the data. …


Why you don’t need to pay much for Learning and gaining Insights

Kerensa Pickett on Unsplash

Whether it’s for university, your job, or simply as input for your next story — there are many interesting sources for free whitepapers and educational material in the field of data. With some sources you have to say that there might be a certain intention to sell a product but with the sources I use, scientific thought is mostly in the foreground. Here are my top places to go:

Everyone knows the for Dummies series. You can buy them on Amazon and in good book stores. Snowflake delivers it for free — after you have registered. Top current topics like…


Benefits of using the Google BigQuery IDE

Photo by Joe Ciciarelli on Unsplash

Work more efficiently with the powerful BigQuery IDE powered by AI that supports Data Engineers, Scientists and BI Developers.

The Chrome Add On features [1]:

- AI engine that optimizes your queries in real-time.

- Adaptive Caching — Never pay twice for the same query.

- Write queries faster with context-aware Smart Compose

- Execute up to 20 queries at the same time.

- Auto-Detect Standard / Legacy SQL.

- Use variables to store values and shorten your workflow.

- Visualize query results with integrated dashboards.

- Download up to 6,000,000 rows to CSV.

You can download the Chrome extension…


Ask the right Questions to succeed in your Data Analytics Projects

Photo by Kalen Emsley on Unsplash

Business Event Analysis & Modelling (BEAM) is an agile requirement gathering for Data Warehouses, with the goal of aligning requirement analysis with business processes rather than just reports. It has its roots in Agile Data Warehouse Design by Lawrence Corr and Jim Stagnitto [1].

The key principles of this concept are [1][2]:

  • Individuals and Interactions: Business intelligence is driven by what users ask about their business. The technical setting is secondary.
  • Business Driven: Well documented data warehouses that take years to deploy will always be out of date. Business users will look elsewhere. …

Christianlauer

Big Data Enthusiast based in Hamburg and Kiel.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store