Lead Data Engineer
Job title: Lead Data Engineer in Los Angeles, CA at WorkHQ Company: WorkHQ Job description: Company ContextSeries A, well-funded US startup in HRTech developing and an AI Recruiter product.This is a US-only, Remote role (Mainland).Role OverviewLead data infrastructure architect managing billions of data points across 250M+ professional profiles.Hire data engineers to aid you in that journey.Core ResponsibilitiesDesign scalable data pipelines processing massive record volumesArchitect ETL processes using PySpark on Amazon EMR (Open to shifting to other solutions like Data Bricks / Snowflake)Distribute enriched data through medallion architecture across Postgres, Athena, OpenSearchIntegrate new data sources into the main pipelineImplement advanced data matching using SplinkTechnical Requirements5-8 years professional data engineering experienceGood proficiency in:PySpark and distributed computingAWS data services (EMR, Glue, Athena)DockerPandas and DataFrame manipulationComplex data format handling (JSONL, Parquet)Strong background in:Big data processing architecturesData warehouse designPerformance optimizationAdvanced Python, SQL skillsNice to HaveProbabilistic record linking expertiseOpenSearch/elasticsearch technologiesMachine learning data pipeline designRecruitment tech ecosystem knowledgeTechnical StackBig Data: PySpark, EMRDatabases: Postgres, OpenSearchCloud: AWSContainerization: DockerData Formats: JSONL, ParquetAnalytics: Metabase, Athena, GlueData Processing: Pandas, SplinkOther ConsiderationsWhile this role has specific requirements - if you lack a few technical skills, but motivated to learn and lead the platform, please apply for consideration.If you are coming from Director/Head of/VP levels that is relevant to this job, you can apply as well.You will need to apply directly on our platform.Thank you for your time. Expected salary: $140000 - 180000 per year Location: Los Angeles, CA Apply for the job now! Apply for this job