Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot Instances
This is the first article in a series that describes how we built a new data-intensive product, AdRoll Prospecting, using an architecture based on Docker containers.
Interested in working with us? We are hiring!
See open positionsThis is the first article in a series that describes how we built a new data-intensive product, AdRoll Prospecting, using an architecture based on Docker containers.
The data science team at AdRoll is constantly working to improve our programmatic bidding algorithm, BidIQ. One recent improvement to BidIQ has been the introduction of a novel modeling technique called Factorization Machines, which is allowing AdRoll to more accurately value every potential impression that comes our way.
In this post, Tim writes about a bash script to run bash scripts.
It has been a long time outstanding to do item to start collecting feedback and net promoter scores from the tens of thousands of customers that AdRoll has. Once at scale it becomes surprisingly challenging to ensure that feedback from users is sent to Product, Engineering and SMB teams.
The AdRoll Growth team was created to ensure that as AdRoll continues to build out more great products for marketers in more regions, we stay true to our founding philosophies. As we ourselves grow, we want the SMB advertisers that make up the core of our business to grow along with us.
By migrating AdRoll’s real-time data pipeline to Kinesis we were able to reduce our end to end latency more than one hundredfold while simultaneously cutting costs and improving system stability. Here we’ll follow architectural decisions, implementation details, and overall learnings from this process.
The AdRoll Engineering team hosts Tech Talks every other Tuesday, where members of our team, and friends, share about a technology, project, or tool they’re excited about. A few weeks ago, we hoted Lee Byron, author of Immutable.js and other great things at Facebook.
AdRoll is building products that allow customers of any size, big or small, with a lot of marketing experience or none, to run high performing marketing campaigns. And with over 20,000 customers in over 150 countries, we’ve been fairly successful in this.
AWS Keys are sensitive cryptographic material that should not be left lying around in code, repositories, or even on developers’ laptops. In EC2, this is solved by assigning Instance Roles and letting the metadata service generate temporary credentials. Outside of EC2, we have solved this using Hologram, an open-source project from AdRoll Engineering.
The advertising industry has undeniably become a data play, as consumers are generating valuable data with every digital interaction. We hear buzzwords like “big data,” “machine learning” and “real-time algorithms,” but little about how these puzzle pieces fit together …