Apache Airflow is a very popular framework for scheduling, running and monitoring tasks, which are grouped into DAG (directed-acyclic graph). Each DAG has several parameters that describe how and when DAG will be executed. DAG itself is composed of tasks arranged in a flow. The DAG parameters are defined as…

How to export datastores from multiple projects using Google Dataflow — with additional filtering of entities. — This is a short extension to my previous story, where I described how to incrementally export data from Datastore to BigQuery. Here, I discuss how to extend the previous solution to the situation where you have Datastores in multiple projects. …

How to employ Google Dataflow to export Datastore to BigQuery with additional filtering of entities — In the last story, I showed how to build a serverless solution to export all kinds from Datastore to BigQuery. The approach presented in that article is completely valid and works for even large datastore. However, the main drawback is that each time we export all rows from the datastore…

Increasing interested in using small computation units enclosed in AWS Lambda, Azure Functions, or GCP Cloud functions brings back the old problem of detecting the termination of the computation task, which is distributed among a vast amount of sub-processes. Let us jump directly into the problem by analysing the following…

It is hard to imagine any serious software development work without continuous integration (CI) and continuous deployment (CD) steps. The first term describes the workflow, where developers integrated their changes frequently to a shared code repository. Each of the integration is preceded by a verification step, which is mostly a…