Open in app

Sign In

Write

Sign In

Jakub Krajniak
Jakub Krajniak

53 Followers

Home

Lists

About

Published in

Towards Data Science

·Pinned

Serverless approach to export Datastore to BigQuery

An easy way to periodically export your Datastore to BigQuery using a serverless approach on Google Cloud Platform — A bit of overview, Google Datastore is a fully managed NoSQL service available in Google Cloud Platform since 2008. It can be treated as a key-value as well as a document database, provides an SQL-like query language and SDK with bindings to many programming languages. The database is organized into…

Serverless

7 min read

Serverless approach to export Datastore to BigQuery
Serverless approach to export Datastore to BigQuery
Serverless

7 min read


Published in

Towards Data Science

·May 2, 2022

Control your Airflow DAGs from an external database

Apache Airflow is a very popular framework for scheduling, running and monitoring tasks, which are grouped into DAG (directed-acyclic graph). Each DAG has several parameters that describe how and when DAG will be executed. DAG itself is composed of tasks arranged in a flow. The DAG parameters are defined as…

Airflow

4 min read

Control your Airflow DAG from an external database
Control your Airflow DAG from an external database
Airflow

4 min read


Published in

Towards Data Science

·Jun 19, 2021

Export Datastores from multiple projects to BigQuery

How to export datastores from multiple projects using Google Dataflow — with additional filtering of entities. — This is a short extension to my previous story, where I described how to incrementally export data from Datastore to BigQuery. Here, I discuss how to extend the previous solution to the situation where you have Datastores in multiple projects. …

Gcp

4 min read

Export Datastores from multiple projects to BigQuery
Export Datastores from multiple projects to BigQuery
Gcp

4 min read


Published in

Towards Data Science

·Apr 5, 2021

Export Datastore to BigQuery using Google Dataflow

How to employ Google Dataflow to export Datastore to BigQuery with additional filtering of entities — In the last story, I showed how to build a serverless solution to export all kinds from Datastore to BigQuery. The approach presented in that article is completely valid and works for even large datastore. However, the main drawback is that each time we export all rows from the datastore…

Gcp

7 min read

Export Datastore to BigQuery using Google Dataflow
Export Datastore to BigQuery using Google Dataflow
Gcp

7 min read


Published in

Nordcloud Engineering

·Apr 28, 2020

Keep track of your cloud computations

Increasing interested in using small computation units enclosed in AWS Lambda, Azure Functions, or GCP Cloud functions brings back the old problem of detecting the termination of the computation task, which is distributed among a vast amount of sub-processes. Let us jump directly into the problem by analysing the following…

Parallel Computing

4 min read

Keep track on your cloud computations
Keep track on your cloud computations
Parallel Computing

4 min read


Published in

Nordcloud Engineering

·Apr 5, 2019

How to optimize deployment to AWS using Bitbucket pipelines

It is hard to imagine any serious software development work without continuous integration (CI) and continuous deployment (CD) steps. The first term describes the workflow, where developers integrated their changes frequently to a shared code repository. Each of the integration is preceded by a verification step, which is mostly a…

Bitbucket Pipelines

6 min read

How to optimize deployment to AWS from Bitbucket pipeline
How to optimize deployment to AWS from Bitbucket pipeline
Bitbucket Pipelines

6 min read

Jakub Krajniak

Jakub Krajniak

53 Followers

problem-solver, programmer, enthusiast of science&tech, PhD

Following
  • Yancy Dennis

    Yancy Dennis

  • ODSC - Open Data Science

    ODSC - Open Data Science

  • Sascha Heyer

    Sascha Heyer

  • Barak Or PhD

    Barak Or PhD

  • TDS Editors

    TDS Editors

See all (156)

Help

Status

Writers

Blog

Careers

Privacy

Terms

About

Text to speech

Teams