Functional Programming Jobs

ML Core Scala/Spark Engineer: Automated Feature Engineering


  • scala
  • spark
  • machine-learning
  • concurrency
  • sql

Quick Facts

San Mateo, CA, US

Job description

Job overview

dotData is hiring high caliber engineers who are excited to democratize Machine Learning with automation. You will work on dotData’s proprietary core engine components: AI-powered feature engineering and AutoML. There’s a lot of interesting stuff to learn and do in building out our high performance parallel distributed platform whether you’re a distributed systems coder, a Machine Learning expert, or both.

  • Our AI-powered Feature Engineering discovers and evaluates millions of combinations of joins and aggregations of relational data. To explore them efficiently we built our own distributed in-memory engine on top of Apache Spark. Skewed keys, wide tables, exploding number of records after join - we have to take care of all of that.

  • Our advanced AutoML component automatically explores and evaluates state-of-the-art ML models with automated hyper-parameter optimization and model selection. Logical plan is divided into computationally-intensive jobs that are executed in parallel with strict resiliency requirements. We take advantage of industry-standard machine learning libraries, like sklearn, XGBoost, LightGBM, TensorFlow, and Pytorch.

If you’re looking to build amazing products with a smart bunch of colleagues and blow the minds of customers this is the place.

Apply now