How to work with BigQuery by using a Python Notebook — Conveniently, using the BigQuery API and thanks to the Python BigQuery library, you can load data directly into BigQuery via Python. Setting up your Environment The integration of libraries via an import command is essential for the communication between BigQuery and Google Datalab. Datalab is the equivalent to a Jupyther Notebook but from Google…

Secure your Data Pipelines within Google Cloud — Along with Python tools and other data services, Google’s Data Transfer is a relatively easy-to-use service to perform data integration and transformations. It can also be used to integrate data sources such as AWS S3. This service has now received an interesting security update. Recap: ETL and ELT with DTS For all those who are not…

How to easily convert the Types in SQL with Parse and Format Date — Quite often in SQL you get to turn a string into a date. Be it during the data integration process or during data analysis. With PARSE_DATE and FORMAT_DATE you have two powerful functions in BigQuery to do so. Here is a small cheat sheet for you. Format Options Here is a list…

How much does an Data Engineer earn? — Even though Data Scientists are currently in great demand and many have this career aspiration, it is the Data Engineer who make the data available for analysis purposes in the first place. But are they paid fairly? How to become a Data Engineer In the meantime, there are many courses with which you can become a…

How, When and Why to Use the BigQuery Function — With the creates table like statement you can copy your table to another with all of the same metadata of the table. This can be useful, for example, if you want to make backup copies or archive data. It’s also nice that you don’t have to do this manually but…

How you can protect your Data — Data protection is becoming an increasingly important issue. Companies want to avoid the risk of horrendous fines [1], however we all personally should handle personal data with confidence, because everyone wants their personal information to be handled responsibly. Tension between Data Protection and Big Data In relation to the topic of Big Data, there is a certain…

How to connect your Python Notebook to MySQL Databases — Often, data scientists or analysts have to access databases for example for ad-hoc analysis. Often, these are MySQL databases, here you can use the MySQL Connector to access a MySQL database from a Jupyter notebook. First step is to install the connector via !pip install command: !pip install mysql-connector-python After…

What are the most followed Tags besides Data Science, Programming, Technology, etc.? — To reach the right audience on Medium, you can tag your articles with up to five topic tags. Besides the perhaps better known tags like: Technology (405.000) Programming (286.000) Javascript (160,000 followers) Data Science (153.000) Python (103.000) There are also many other popular tags you can use when writing about…

Use Session User in BigQuery — With the Session User function you can develop a user access concept in BigQuery via SQL relatively easily. Here is a short tutorial. SESSION USER The function returns the email address of the user executing the query. Example [1]: SELECT SESSION_USER() as user; Realize the Access Concept Step 1: Firstly, one piece of information: The user…