Tabular Data Transferring Patterns
February 05, 2023I want to discuss the work that goes into engineering a data pipeline. And how this crucially depends on the choices the data provider makes.
By Philipp Jung, a data engineer with one foot still in academia. Follow me on mastodon.world/@pljung, or reach out on LinkedIn.
I want to discuss the work that goes into engineering a data pipeline. And how this crucially depends on the choices the data provider makes.
As a beginner rock climber, I experienced the scariest moment of a climb at its very end. That is, after I clipped a quickdraw into the first bolt. Which serves as a securing point and prevents a fall to the ground. It happened after I climbed the crux, which is what rock climbers call the hardest section of a route. It happened at the very top of the route, where I found a couple of bolts connected via a metal chain. This last anchor marks the end of the climb.
Over the last year I experimented with different IDEs. One of my tasks at work is maintaining a react application, which tempted me to use VS Code for developing it - writing Typescript in an IDE written in JavaScript felt like a good fit.
There is that workflow where a developer 1) encounters a problem, 2) googles the problem and 3) opens links until they find something they can copy-paste to solve the problem. I'm not the first one to discuss this. The pattern has been identified in the past, and it has been named The Stack Overflow Antipattern. Knowing how to use a search-engine to solve a problem is an essential skill for any knowledge worker in 2022. And despite its name, the Stack Overflow Antipattern is quite powerful.
A rough guide for establishing a backup routine on a machine running Fedora 34.
With early vaccination data from the Robert Koch Institute, I run a linear regression to get an estimate to predict when the Germans will reach herd immunity.
JavaScript Objects and Arrays are mutable data structures, and constants and variables are references. I tripped over this recently working on state update logic in a React app.
Some months ago I gave a talk at Beuth University of Applied Sciences about the application of JavaScript as a fullstack language. I built a with an express backend and a react frontend and gave a demonstration. It was not long before people started asking questions about CORS, global state and lifting state up.
One of the biggest tasks to keep me busy in 2019 was research. More precisely, my work for Prof. Felix Biessmann on a new approach to Relaxed Functional Dependency (RFD) detection, and in this article I want to present and discuss that work.
I see a lot of tools and guides on how to create backups in Linux, but they often lack an explanation of the key principles *behind* the backup. In this post, I want to explore the basics, and demonstrate how and why Linux backups can be effectively employed.
Bad news first. Python is a poor choice for concurrent programming. A principal reason for this is the 'Global Interpreter Lock' or GIL. The GIL ensures that only one thread accesses Python objects at a time, effectively preventing Python from being able to distribute threads onto several CPUs by default.
Scientific code is bad. Programming it often serves as a tool to bridge the gap between mathematical theory and predictive model. As a result, scientific code is frequently of bad quality. I'll show you how to get away with it.
I have been asked repeatedly by friends and colleagues how I learned programming. Having done my Bachelor's Degree in Physics and currently working towards a Master's in Industrial Engineering and Project Management, I do not have an academic Computer Science background at all.
"Sie müssen 30 ECTS-Punkte machen. Dabei können Sie jeden Kurs einbringen, der an einer Berliner Hochschule angeboten wird." So oder so ähnlich erklärt mir mein Studiengangsleiter an der Beuth-Hochschule die Auflage, an die meine Freigabe zur Masterarbeit gebunden ist.'
Im Rahmen der Founders Factory besuchte ich an einem grauen Mittwochmorgen das St. Oberholz am Rosenthaler Platz. Mitgründer und Namensgeber des ersten Coworking-Spaces Berlins, Ansgar Oberholz, empfing unsere Gruppe und gab einen Vortrag mit anschließender Diskussion.
Smartphones sind cool. Taschencomputer, mit denen man Fotos aufnehmen und telefonieren kann und die einem den Weg zur nächsten U-Bahn-Haltestelle zeigen. Seitdem das erste iPhone 2007 auf den Markt kam, haben die schlauen Telefone die Welt verändert.
Einer der Vorzüge, die das Großstadtleben mit sich bringt, ist die Möglichkeit, die urbane Infrastruktur zu nutzen. Und die verändert sich in rasender Geschwindigkeit. 2012 habe ich zum ersten Mal den Fahrradverleihdienst der Deutschen Bahn vor der TU Berlin gesehen.
Im Herbst letzten Jahres habe ich einfach nicht aufgehört, Fahrrad zu fahren. Bis dahin habe ich immer mit dem ersten Herbstregen das Fahrrad in den Keller gestellt und meine Verbindungen bei der BVG nachgeschaut. Letztes Jahr war das anders. Egal, ob es morgens stockdunkel war, geregnet oder geschneit hat, ich bin auf mein Rad gestiegen und zur Uni gefahren. Und das hat auch einfach so geklappt.
Gerade liegt mein Urlaub hinter mir. Nach etwas mehr als einer Woche Aktivurlaub in den Bergen bin ich froh, wieder einen freien Blick zum Horizont zu haben. Auf dem Weg zurück nach Berlin lege ich einen Zwischenstopp in meiner Heimatstadt Bad Kreuznach ein, um Freunde und Familie zu besuchen. Vom diesjährigen Rekordwetter motiviert, mache ich eines morgens einen Abstecher in das Salinenbad.