Big Data Engineer

  • Development
  • Irvine
  • Posted 1 month ago


Powering our industry leading services requires highly scalable, available, reliable, secure, and performant systems. Our cloud and platform engineering teams build the infrastructure for our web crawling services. From Kubernetes to Machine Learning we are continually striving to push the envelope to bring the best value to our customers.  At PriceSpider we are looking for enthusiastic and passionate engineers to join our team.  If you love working to bring real world value to many of the world’s largest companies, we would love to hear from you.  Our platform runs thousands of jobs and processes millions of requests each day as we strive to collect the most accurate information about the way our customers products are used and consumed around the globe.

We are looking for a Big Data Engineer to be a part of our Software Engineering team. In this role, you will be delivering software and striving for operational excellence for our sophisticated web crawling solutions and the processing pipeline. This is a unique opportunity to learn and work with other talented engineers working in a dynamic environment leveraging many of the latest technologies in the industry.

Essential Functions:

  • Spend your time doing 20% architecture and design, 60% ETL and data cleansing, 20% big data analysis
  • Enhancing, maintaining and curating our big data platform to enable our data analysts to productively work with extracted and curated data within the Hadoop and Google Bigquery environment
  • Respond to requests to pull data from various sources into usable form.

Required Education and Experience:

  • BS/MS in Computer Science or Mathematics, equivalent work experience
  • You have 6+ years of experience in developing enterprise level software
  • A solid understanding of Object Oriented Programming, Database Operations and Data Quality engineering
  • Strong understanding of high traffic environments with at least 1 terabyte of data
  • You are proficient in Java or Scala, Python, Node.js, regular expressions
  • You are proficient in SQL, OLAP, Hadoop, HDFS, Hive, Spark, Hbase
  • You have experience with ETL, Data Cleansing, ERD design, Data design, data performance optimization
  • Experience developing in an Agile/SCRUM environment preferred

PriceSpider is an Equal Opportunity Employer.  Local candidates only.

Maximum file size: 50 MB.