Senior Software Engineer / Data Engineer – Cyber Threat Intelligence Collections

About the Role

We are looking for a Senior Software Engineer / Data Engineer to join our Threat Intelligence Collections team. You will design, build, and operate large-scale data collection systems that gather, enrich, and deliver high-fidelity threat intelligence from the open web, deep web, and dark web sources. This is a high-impact, technically challenging role at the intersection of cybersecurity, data engineering, and modern AI-driven automation. You will own the full lifecycle of threat data pipelines — from raw source ingestion to clean, structured, and queryable intelligence that powers our detection, hunting, and response capabilities.

Key Responsibilities

• Design and scale high-volume data collection systems that ingest threat data from open-web, deep-web, and dark-web sources (including Tor, I2P, and other anonymity networks).
• Build and maintain robust, fault-tolerant data pipelines (batch + streaming) for ingestion, transformation, enrichment, and storage.
• Develop advanced data parsing and normalization logic for unstructured, semi-structured, and rapidly evolving threat data formats
• Design, build, and maintain agentic workflows — autonomous, LLM-powered agents that can reason, adapt, and execute multi-step collection and enrichment tasks.
• Architect and operate high-scale media pipelines for ingesting, processing, storing, and enriching large volumes of media content (images, videos, screenshots, documents, etc.) collected from threat intelligence sources.
• Design and build internal and external APIs that expose collected intelligence to downstream security teams and platforms.
• Write clean, production-grade Python code as the primary language for all data engineering and automation work.
• Collaborate closely with threat intelligence analysts, detection engineers, and security researchers to translate intel requirements into reliable, scalable data systems.
• Monitor, troubleshoot, and continuously optimize collection performance, data quality, and latency at scale.
• Stay current with emerging threat actor TTPs, data sources, and evasion techniques to proactively improve collection coverage.

Required Qualifications

• 5+ years of experience as a Software Engineer or Data Engineer, with a strong focus on large-scale data collection and processing.
• Deep experience collecting data at scale from open and dark web sources (scraping, crawling, API integration, Tor-based collection, anti-detection/anti-bot techniques).
• Strong expertise in building and operating data pipelines (Airflow, Dagster, Prefect, Spark, Kafka, Flink, or equivalent).
• Advanced data parsing skills — experience turning messy, unstructured web data into clean, structured intelligence.
• Expert-level Python proficiency (including async, concurrency, and performance optimization).
• Hands-on experience designing and implementing agentic workflows (LangChain, CrewAI, AutoGen, LlamaIndex, or similar frameworks) and integrating LLMs into production data systems.
• Track record of building and maintaining production APIs (FastAPI, Flask, or similar).
• Strong understanding of cybersecurity concepts and threat intelligence data types (IOCs, TTPs, malware samples, phishing kits, underground forums, etc.).
• Comfortable working in a fast-paced, ambiguous environment with high-stakes data quality requirements.

Preferred Qualifications (Nice-to-Haves)

• Experience with cloud platforms.
• Knowledge of modern data stack tools (dbt, Snowflake, ClickHouse, DuckDB, etc.).
• Familiarity with containerization (Docker, Kubernetes) and infrastructure-as-code.
• Prior work in cybersecurity, threat intelligence, or intelligence collection organizations.
• Contributions to open-source tools or public research in the threat intel or data collection space.

What We Offer

• Competitive compensation and equity
• Comprehensive health insurance (medical, dental, and vision)
• 401(k) retirement plan
• Unlimited PTO — take the time you need to recharge and maintain work-life balance
• Generous learning & development budget for courses, conferences, books, certifications, and tools to support your continued growth
• Opportunity to work on cutting-edge threat intelligence systems that directly protect organizations and people
• A collaborative, high-trust engineering culture that values ownership and innovation

Apply tot his job Apply To this Job