In the field of Data Analytics and related topics like BI, Data Science, Data Engineering etc. you often will hear about the same problems when working in a project or on a product. Here, I want to share my experiences and possible solutions.
One of the most unpleasant moments in the life of every project or product manager is when the business department complains about the data quality. The problems can be of different nature. Errors in the source system, ETL process or in the report.
Solution: Here, it is a good idea to set up a monitoring system and…
Personal data is the core concept of data protection. Data protection law only applies when data relates to individuals. The GDPR for example increases fines to up to 20 million euros or, in the case of large companies and groups, up to 4% of the global group turnover of the previous year [1]. When working in the field of Big Data, Data Science or related fields it is essential to know about these laws and how anonymization and pseudonymization give the possibility of still using the data for your use cases.
This is any information relating to an identified or…
In the world of Big Data, data visualization tools and techniques are essential to analyze large amounts of information and make data-driven decisions as data is increasingly used for important management decisions. So there is a trend away from gut feeling and emotional decisions towards rational choices that are made based on numbers. Therefore, reports and visualizations have to be easily understood and meaningful.
It is increasingly beneficial for professionals to be able to use data to make decisions and visuals to tell stories that communicate how data informs the question of person, subject, time, place, and method [1]. In…
The File Transfer Protocol is for the communication of people and devices over the Internet and other networks works through protocols [1]. Because FTP is an older method of data transfer, such transfers are compatible with many legacy and/or on-premises HR and business systems, making it a useful option if you want to integrate an older system with newer, cloud-based software.
In the past, most digital systems were connected via FTP integration, where one system exports data in a “flat file” format (often a spreadsheet) and another system imports the data. …
Whether it’s for university, your job, or simply as input for your next story — there are many interesting sources for free whitepapers and educational material in the field of data. With some sources you have to say that there might be a certain intention to sell a product but with the sources I use, scientific thought is mostly in the foreground. Here are my top places to go:
Everyone knows the for Dummies series. You can buy them on Amazon and in good book stores. Snowflake delivers it for free — after you have registered. Top current topics like…
Work more efficiently with the powerful BigQuery IDE powered by AI that supports Data Engineers, Scientists and BI Developers.
The Chrome Add On features [1]:
- AI engine that optimizes your queries in real-time.
- Adaptive Caching — Never pay twice for the same query.
- Write queries faster with context-aware Smart Compose
- Execute up to 20 queries at the same time.
- Auto-Detect Standard / Legacy SQL.
- Use variables to store values and shorten your workflow.
- Visualize query results with integrated dashboards.
- Download up to 6,000,000 rows to CSV.
Business Event Analysis & Modelling (BEAM) is an agile requirement gathering for Data Warehouses, with the goal of aligning requirement analysis with business processes rather than just reports. It has its roots in Agile Data Warehouse Design by Lawrence Corr and Jim Stagnitto [1].
The key principles of this concept are [1][2]:
In this article I want to demonstrate how you can use open data for interesting use cases. If you are interested in how to get started with BigQuery and Alteryx, this article might be the one for you. In this walkthrough, I will use OpenStreetMap data from BigQuery. It’s great that Google published it for free in their public data sets. So you can easily query geo information with SQL [1]. At the end of the article I have listed other interesting public data sources that you can use for your projects.
One important side note: Google uploaded their data…
In the field of logistics and warehousing, there is of course a lot of software that helps you with the optimal route, warehouse utilization and supply chain management. But what are the basic KPIs in this area? Whether you are a data scientist, analyst or business analyst, it is useful to understand what the individual departments actually do. What are their goals? How do they earn their money directly or indirectly? How can the success be measured? These question with their respective answers helped me to understand the key figures of the individual departments.
The first question is of course…
Here is everything you have to know about Artificial Intelligence, Machine Learning and Deep Learning.
Artificial Intelligence is an umbrella term and describes the broad approach of using machines to imitate intelligent human behavior in order to solve problems.
Big Data Enthusiast based in Hamburg and Kiel.