Junior Data Engineer with DevOps share (Apache Nifi, Pulsar, AirFlow, Grafana, JupyterLab, Python, Linux, SQL, Azure, AWS) Remote/Frankfurt
Job description:
**63 of which 5 PT onsite** The project “Automatic Disturbation on the Freight Car” (ASaG) is intended to examine whether the image data of freight cars can be tested with regard to damage by means of artificial intelligence (AI) and this AI can be used in operation. Whether and how work processes would have to be adapted to production and could be tested in the project. Every day, 1400 cars with about 40 images are recorded per camera bridge. The most important camera bridge for ASaG is Munich Nord, where further sensor technology is to be used. The AI must show its suitability in the test mode near production with these data sources. **The tasks are:** * Participation in migration of MLOps pipelines to Azure/BuildIT platforms * Implementation, concepts * Developing ETL processes in the Big Data environment according to the requirements of partners in an agile team * Use case specific and overall * Operations, Trouble Shooting, Workarounds and Accompanying AI-Use-Case Operations Testing in existing open source MLOps AIC stack Basic requirements - if this part does not run, the rest is automatically reset (covering usually low but top priority) **MUSS Requirement:** * Big Data Tool Stack (min. basic knowledge, comprehensible based on work experience and project references, indication of knowledge (basic knowledge, extensive experience, proven expert) Apache Nifi, Apache Pulsar, Apache AirFlow, Grafana, JupyterLab * Advanced knowledge Python (Practical experience from projects Recognisable on the basis of project history) * Advanced knowledge of Linux (Practical experience from projects Recognisable based on project history) * Knowledge of database (SQL) technologies (practice experience from projects). Descriptive on the basis of the project history.) * Azure (min. basic knowledge) **SOLL Requirements:** * Big Data Tool Stack (min. Basic knowledge): Apache Hadoop / HDFS , Apache Drill (Experience based on work experience and project references, knowledge indication, basic knowledge, extensive experience, designated expert) * AWS Cloud * Ability to act and communicate in an agile project environment (assessment of team leaders and past customers) * Responsibility (assessment of team leaders and past customers) * Flexibility & adaptability (assessment of team leaders and past customers) * Criticism (open direct communication) (assessment of team leaders and past customers) * Interest in new project experiences (assessment of team leaders and past customers) * Get to know rail freight