We’ve had an incredible group of people join Scale—and we’re still <a href="https://scale.com/about#jobs" rel="noopener noreferrer" target="_blank">hiring</a>! We wanted to take a moment to follow up on <a href="https://scale.com/blog/scale-engineering-team" rel="noopener noreferrer" target="_blank">this post</a> and call out the latest team members that have joined the Engineering, Product and Design teams at Scale: <ul><li> </li><li>Catherine Wu / Catherine is a full-stack engineer at Scale working primarily on our infrastructure team. She has previously worked as an engineer at Google and product manager at Yelp. She studied computer science at Princeton University, where she also competed on the synchronized skating team.</li><li> </li><li> </li><li>Eugene Fratkin / Prior to Scale, Eugene led R&amp;D for the Cloud division at Cloudera, responsible for the development of managed distributed data processing services in AWS/Azure. Prior to Cloudera, Eugene was a co-founder and VP of R&amp;D at Lifecode – a Sequoia Capital-backed genomic cancer diagnostic start-up. Prior to Lifecode, he was one of the founding members of Greenplum’s machine learning team, which started the MADlib Apache project. Eugene holds a PhD in Machine Learning from Stanford. Eugene also served as an assault team section leader in the United States Marine Corps for missions such as the anti-terrorism quick reaction force within the continental US, and the initial phase of Operation Iraqi Freedom.</li><li> </li><li> </li><li>Gabriel Andretta / Gabriel previously worked at <a href="https://auth0.com/" rel="noopener noreferrer" target="_blank">Auth0</a> with Ricky, Martín and Ramiro, where he rebuilt <a href="https://github.com/auth0/lock" rel="noopener noreferrer" target="_blank">Lock</a> from scratch and was the lead developer of the browser SDKs. He was a contributor to the sinatra web framework, a coach in the first edition of <a href="https://twitter.com/ClojureBridgeBA" rel="noopener noreferrer" target="_blank">Clojure Bridge Buenos Aires edition</a>, and built a <a href="https://github.com/gnandretta/literate-2048" rel="noopener noreferrer" target="_blank">2048</a> clone with almost no mutable state.</li><li> </li><li> </li><li>Steven Salka / <a href="http://ssalka.io/" rel="noopener noreferrer" target="_blank">Steven</a> joined the engineering team at Scale as a product-focused full-stack engineer, working largely on internal applications and the frontend for Scale’s LiDAR tool. Previously, he was the first engineering hire at <a href="https://podible.co/" rel="noopener noreferrer" target="_blank">Podible</a>, and before that built out a UI framework for text-based annotations at <a href="https://x.ai/" rel="noopener noreferrer" target="_blank">x.ai</a>. Steven studied Applied Mathematics at UC Berkeley.</li><li> </li><li> </li><li>Ramiro Silveyra d'Avila / Prior to Scale, Ramiro was a software engineer at <a href="https://auth0.com/" rel="noopener noreferrer" target="_blank">Auth0</a> working on the marketing and engineering teams. He loves open source software, especially web tooling. He has contributed to yarn, babel, stylelint, eslint plugins and webpack loaders.</li><li> </li><li> </li><li>Michael Woo / Michael is a supply product manager at Scale. After graduating from Harvard in Computer Science, he joined McKinsey &amp; Company as a business analyst. Michael is a former captain of the Harvard Fencing Team and National Junior Team Member for the US National Fencing Team.</li><li> </li><li> </li><li>Sara Xiang / Sara is a supply product manager at Scale. She previously worked at Twitter on the Moments team, and at Facebook on the Oculus Developer Technologies and Oculus TV teams. She studied computer science at Carnegie Mellon University.</li><li> </li><li> </li><li>Pato Perez / Pato is a full-stack engineer at Scale working primarily on our LiDAR annotation tool. He previously worked as an Android and iOS engineer at other startups. He was the winner of multiple software engineering competitions, including Microsoft Imagine Cup @ Australia.</li><li> </li><li> </li><li>Shivaal Roy / Shivaal graduated from Stanford with a B.S. and M.S. in computer science. He founded Stanford's health hackathon, <a href="http://healthplusplus.stanford.edu/" rel="noopener noreferrer" target="_blank">Health++</a>, and was president of <a href="http://shift.stanford.edu/" rel="noopener noreferrer" target="_blank">SHIFT</a>, Stanford’s health-tech student organization. Prior to Scale, he worked at Google on the Photos Machine Intelligence team, and at Oscar Health.</li><li> </li><li> </li><li>Mauro Giamberardino / Mauro is a senior full-stack engineer who previously worked for companies like Globant and Spark Digital. He studied Systems Engineering at Buenos Aires University.</li><li> </li><li> </li><li>Martin Cabral / Martin joins Scale as a software engineer with ten years of experience in different positions and companies. His last adventure was as a product manager at <a href="https://www.auth0.com/" rel="noopener noreferrer" target="_blank">Auth0</a> where he learned to work very closely with customers and understand how they approach products. When he is not working, you can find him gaming (Dota2, CS, Fortnite et al.) or playing real life tennis.</li><li> </li><li> </li><li>Nathan Hayflick / Nathan is a frontend engineer at Scale. He joins from Uber, where he worked for 3 years on the Safety Engineering and Develeper Platform teams, and worked at Rockbot before that. He studied Literature at Stanford University.</li><li> </li><li> </li><li>Jeff Hilnbrand / Jeff is the Design Manager at Scale. He was most recently a Senior Product Designer at Facebook on the Groups and Payments teams. Prior to that, Jeff founded a design agency called Minimill and worked on design teams at Autodesk and Venmo. Jeff studied product design and engineering at the University of Maryland.</li><li> </li><li> </li><li>Andrew Wang / Andrew was a tech lead at Cloudera working on HDFS and cloud-based machine learning platforms for 6 years. He’s one of the biggest contributors to the open-source Apache Hadoop project, served as the release manager for Apache Hadoop 3.0, and is an Apache member. Before that, he was a PhD student in distributed systems at UC Berkeley and studied computer science at the University of Virginia.</li><li> </li><li> </li><li>Craig Weiss / Craig joins Scale with experience from Lyft, Snapchat, Facebook, and Google. Craig frequently competed in and won several awards at regional hackathons and competitive programming competitions (including the ACM-ICPC). Craig studied computer science at the University of Maryland.</li><li> </li><li> </li><li>Aatish Nayak / Aatish is a supply product manager at Scale, with previous experience at Shield AI, Uber, MongoDB, and Skurt. At ShieldAI he worked on life-saving autonomous robots for special operations and law enforcement use cases. Aatish studied computer engineering at Carnegie Mellon University and led Autolab, a startup for auto-grading CS homework assignments.</li><li> </li><li> </li><li>Ty Wilkins / Ty has worked closely with Scale from the beginning to establish Scale’s branding, icon system, and illustration style. Previously, Ty was an early employee at <a href="https://dribbble.com/shots/2994377-Robinhood-Gold" rel="noopener noreferrer" target="_blank">Robinhood</a> where he created product UI, branding, illustration, and marketing. He has extensive experience in branding, working with clients including Google, <a href="https://dribbble.com/shots/2347496-Facebook" rel="noopener noreferrer" target="_blank">Facebook</a>, and <a href="https://dribbble.com/shots/195804-Target-Education" rel="noopener noreferrer" target="_blank">Target</a>. Ty has also branded numerous startups including <a href="https://auth0.com/" rel="noopener noreferrer" target="_blank">Auth0</a>, <a href="https://www.useloom.com/" rel="noopener noreferrer" target="_blank">Loom</a>, and <a href="https://dribbble.com/shots/314295-Orchestra" rel="noopener noreferrer" target="_blank">Orchestra</a> in addition to many open source projects including <a href="https://dribbble.com/shots/4738771-Celery" rel="noopener noreferrer" target="_blank">Celery</a> and <a href="https://dribbble.com/shots/2153014-JWT" rel="noopener noreferrer" target="_blank">JWT</a>. His editorial illustration work has been featured in publications including <a href="https://dribbble.com/shots/192949-Monocle" rel="noopener noreferrer" target="_blank">Monocle</a>, <a href="https://dribbble.com/shots/1215603-Bloomberg-Markets" rel="noopener noreferrer" target="_blank">Bloomberg Markets</a>, <a href="https://dribbble.com/shots/2083018-ESPN" rel="noopener noreferrer" target="_blank">ESPN Magazine</a>, and <a href="https://dribbble.com/shots/766236-Wired-UK" rel="noopener noreferrer" target="_blank">Wired UK</a>. Ty holds a B.F.A. in Graphic Design from Auburn University.</li><li> </li><li> </li><li>Facundo Pepe / Facundo is a UI Developer in charge of the marketing landing pages at Scale. He previously worked at Infobae, the most popular news media in Argentina and at <a href="https://aerolab.co/" rel="noopener noreferrer" target="_blank">Aerolab</a>, a digital agency where he and his team won a <a href="https://www.a24.com/actualidad/a24com-gano-un-martin-fierro-digital-por-la-aplicacion-juga-24-12112018_H1xcRshkN" rel="noopener noreferrer" target="_blank">Martín Fierro Digital award for Best New Application</a> with a betting game for the World Cup. Better known as Turbo because he makes things fast, Facundo loves to deliver pixel perfect projects while working on user interactions that look cool and also perform well.</li><li> </li></ul>

Scale's Engineering, Product and Design Teams - Meet Our Team

Scale Engineering, Product and Design Teams

Scale AI is pleased to announce the full release of <a href="https://www.nuscenes.org/" rel="noopener noreferrer" target="_blank">nuScenes by Aptiv</a> —a large-scale open source dataset for autonomous driving. Scale AI was involved as Aptiv's data labeling partner on this initiative. On March 28, 2019, Aptiv announced the full dataset release of nuScenes, an initiative intended to broadly support research into computer vision and autonomous driving by AV innovators and academic researchers to further advance the mobility industry. One key bottleneck in the advancement of deep learning is access to labeled data. Most machine learning algorithms learn from large amounts of examples to understand the world. These incredible algorithms are powered behind the scenes by billions of hand-labeled examples of what humans can easily understand. While we like to believe as computer scientists that it is all about the algorithms, labeled data is incredibly impactful to progress. The current age of deep learning was ignited by the launch of the ImageNet dataset by Fei-Fei Li's lab at Stanford in 2009. As Fei-Fei said once in an interview: <blockquote>Our vision was that Big Data would change the way machine learning works. Data drives learning.</blockquote><blockquote> </blockquote>The release of the full 1000 scenes means nuScenes now includes 1.4M camera images, 390k LiDAR sweeps, 1.4M RADAR sweeps and 1.4M object bounding boxes in 40k keyframes. nuScenes is the largest multi-sensor dataset for autonomous vehicle data with LIDAR, RADAR, camera, IMU, and GPS, and builds upon great work in the field by others such as the KITTI dataset. Collected in Boston and Singapore, nuScenes data contains 7x more object annotations than KITTI and will enable researchers, students and developers to study challenging urban driving situations. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://lidar-now.scale.com/examples/demo" height="100%" width="100%"></iframe> Sensor Fusion Viewer From conception, we partnered with Aptiv to produce the nuScenes dataset. By leveraging Aptiv's autonomy stack and <a href="https://scale.com/sensor-fusion-annotation" rel="noopener noreferrer" target="_blank">our sensor fusion and LIDAR annotation products</a>, nuScenes sets a new standard for quality in public datasets, along with a web-based visualizer for LIDAR and camera data for <a href="https://www.nuscenes.org/explore/scene-0120/0" rel="noopener noreferrer" target="_blank">exploring the dataset</a>. It is crucial to machine learning development to be able to visualize data to develop intuitions about the problems at hand. We, <a href="https://eng.uber.com/avs-autonomous-vehicle-visualization/" rel="noopener noreferrer" target="_blank">like</a> <a href="https://medium.com/cruise/introducing-worldview-749aaf98112d" rel="noopener noreferrer" target="_blank">others</a>, noticed a gap where previous options like <a href="http://wiki.ros.org/rviz" rel="noopener noreferrer" target="_blank">rviz</a> were not built for a web-based world. Our LiDAR visualizer allows point cloud data to be easily embedded into any webpage and shared. We've also built it to be developer-friendly and usable for a broad range of use cases. We hope that the nuScenes dataset will become the new standard in the industry and allow researchers to develop safer autonomous driving technology. There is a large gap today in the machine learning community—the larger organizations have significantly larger datasets and compute budgets than smaller organizations or universities. Our hope is to bridge that gap and democratize access to state-of-the-art data in autonomous vehicles. With the launch of nuScenes, we hope to do for the autonomous vehicle community what ImageNet did for computer vision.

nuScenes by Aptiv ImageNet for Autonomous Vehicles

Scale is committed to accelerating the development of AI applications and serving customers with high-speed, accurate, and affordable data annotation. As part of this mission, Scale and Ouster are pleased to announce a data integration partnership that makes it significantly easier and faster for Ouster customers to send their LiDAR data to the Scale platform for annotation. The two companies have worked together for months to establish a common data format and labeling process that combines the image-like output from Ouster’s lidar sensors and the industry-leading capabilities of Scale’s Sensor Fusion annotation (annotation for LiDAR and RADAR) and Semantic Segmentation annotation solutions to save perception teams time and work. The new approach removes some of the most difficult and time-consuming aspects of data labeling for companies working on everything from autonomous vehicles to industrial robotics and drones. The partnership has a number of specific advantages for customers, including: <ul><li>Streamlined data transfer - A standardized, efficient data</li><li> format reduces the amount of work customers must perform prior to sending</li><li> data for annotation.</li><li> </li><li>No sensor fusion required - Ouster’s lidar sensor outputs</li><li> 2D camera-like imagery in addition to 3D data, reducing the time spent on</li><li> calibrating cameras.</li><li> </li><li>Automatic 2D Mask Generation - Customers can now send a</li><li> single image with 2 API calls to Scale and receive both 2D instance and</li><li> semantic segmentation masks and 3D bounding boxes.</li><li> </li><li> </li></ul><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8fdb9340-ff9b-47c0-8ac0-b9811d03c400/public" alt="Point cloud computed only from depth information encoded in image pixels."> Point cloud computed only from depth information encoded in image pixels. Ouster and Scale are leading the industry with this first-of-a-kind integration. In the coming months, Scale intends to form similar data integration partnerships with a broad ecosystem of sensor manufacturers to continue to improve customer experience for LiDAR and RADAR annotation.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/ec025881-9915-4b23-e38c-b5d13ac21300/public" alt="Full 360-degree camera view helps taskers locate and annotate objects."> Full 360-degree camera view helps taskers locate and annotate objects. These partnerships will reduce the amount of work customers have to put in to send their data to Scale.AI, to develop a more seamless data pipeline.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/91d02897-d750-45b9-96e3-8867cf5e8700/public" alt="Accurate projections of cuboids from 3D to 2D.">Accurate projections of cuboids from 3D to 2D.

Scale is committed to accelerating the development of AI applications and serving customers with high-speed, accurate, and affordable data annotation.

Announcing the Scale and Ouster Data Integration Partnership - Scale

Announcing the Scale and Ouster Data Integration Partnership

At Scale, we pride ourselves in developing superior tooling and implementing forward-looking solutions for problems our customers on the cutting edge of computer vision, including self-driving cars, AR/VR, and drones encounter. For example, our 2D image annotation endpoints now support both per-annotation attributes and label hierarchies, allowing you to get richer labeled data out of Scale.This post will briefly outline the differences between the two. <h2>Per-Annotation Attributes</h2> Defining per-annotation attributes for a task helps you extract additional structured information about individual annotations, beyond what a single selected label can provide. The attribute interface can also adapt dynamically to previously-selected attribute choices. For example, only if workers answer “No” to “Is the car currently parked?” will they be asked “ Which direction is the car heading?” If they answer “yes” to “Is the car currently parked?” the interface will skip the question “Which direction is the car heading?”<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/ee91d053-5b42-42f4-32e8-a819c3849300/public" alt="Dashboard selection"> <h2>Label Hierarchies</h2> Nesting labels can also help you extract additional structured information from your data. By default, 2D annotation tasks sent to Scale have a limit of 7 flat labels to reduce the cognitive load on our human labelers and keep their accuracy high. But using subchoices, a worker can, for example, first categorize an object as pedestrian or vehicle, and based on that choice select the specific type of pedestrian or vehicle.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2a97419c-f9e2-492b-c46d-34c0e8528a00/public" alt="Dashboard selection"> Ready to get started? Take a look at our <a href="https://docs.scale.ai/#annotation-attributes" rel="noopener noreferrer" target="_blank">Annotation Attributes</a> documentation for how to include attributes in your tasks, and for how to use attribute conditions to dynamically request specific attributes. Or see our <a href="https://docs.scale.ai/#nested-annotation-labels" rel="noopener noreferrer" target="_blank">Nested Labels documentation</a> to learn how to specify large lists of labels on annotation tasks.

At Scale, we pride ourselves in developing superior tooling and implementing forward-looking solutions for problems our customers.

Per-Annotation Attributes & Labelling for Image Annotations

Metadata to Enrich Image Annotations

At Scale, we are excited to help you solve your most challenging computer vision problems. One important and well-studied problem within computer vision is that of semantic segmentation, which aims to understand images at the pixel level. More precisely, in semantic segmentation, we want to understand which semantic class each pixel of an image belongs to. Below we show a street scene, semantically segmented into a small set of class labels including vehicles, pedestrians, and road markings:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/b3f2baa7-c352-4499-b4db-f024cd404a00/public">A street image<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/39d76976-0384-47be-705e-55b47e10ee00/public">The image segmented into semantic classes Supervised approaches trained on large datasets of pixel-wise annotations are currently the best at predicting semantic classes in several public benchmarks and challenges. Unfortunately, obtaining accurate pixel-wise training data can be incredibly time-consuming; each pixel of an image must be manually assigned a label by a human, usually through the use of Photoshop-like tools. To provide our customers with the fastest turnaround times for semantic segmentation, Scale has added a “Guided Automatic Segmentation” feature to our segment annotation tool. We hope to make the process of creating pixel annotations faster and easier by allowing our labelers to provide only a few points of a segment and automatically filling in the rest. Obtaining segment annotations by clicking a few key boundary points! <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/9971da18-0d5a-4387-d732-2a126f343900/public"> Segmenting a car with four extremal points as human input. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/e132db34-0b30-4d48-982d-af66f1afd100/public"> road label with four extremal points as human input. Our new semi-supervised tool allows us to empower our labelers with insights from current data-driven methods. Instead of attempting to segment the entire image from scratch, our labelers must only click on at least four extremal points of any object. By reducing our area of interest from the whole picture to a smaller single-class semantic region, we can use the convolutional features from the image data augmented by the chosen extremal points to create a proposed pixel map, which can then be further fine-tuned at the labeler’s discretion. Our Guided Automatic Segmentation tool ultimately allows our workers to label pixel-wise annotations much faster with higher quality, so we can continue to label all of your semantic segmentation data at best-in-class accuracy and cost. We are excited to continue exploring other ML techniques to improve all of our APIs -- stay tuned for more blog posts coming soon!

AI-Assisted Semantic Segmentation Labeling

Introducing AI-Assisted Semantic Segmentation at Scale

Scale relies on Mode Analytics for internal SQL and Python based reporting and analytics, helping power our operations, insights and business decisions. Many of these reports we set to run on a schedule, so that we can always go to a report and see data that’s no more than fifteen minutes to an hour out of date. However, Mode doesn’t natively support a method of notifying people when something needs attention, e.g. when our margins on a project take a sharp dive or many new tasks are suddenly created. We’ve set up a system to utilize these reports to easily alert us when something’s unusual, so that someone can take action.<h2>Alert generation</h2> <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Ff0583e4d10994410bb5db605266bf655" alt="Scale and Slack Dashboards"> The flow for alerting goes as follows: when a report is run (either manually or on a schedule), it sends a webhook to an AWS Lambda function, which then pulls the report data from the Mode API, and retrieves a list of alert definitions from our database. It then looks through the alert definitions and sees if any are applicable to the current report being run, and if so, creates an alert for each row in the report that satisfies conditions set out in the matching alert definition. These alerts are then sent to a Slack webhook, which generates an alert in a channel specified in the alert definition.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2b8f800f-0c2f-4a52-53d5-d2a3205d8100/public" alt="Slack Dashboard"> <h2>Defining alerts</h2> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8a14c54f-48ce-48b0-0d62-dbe006402700/public" alt="Alerts Dashboard"> Alerts are defined in our database, and are easily viewable/creatable from our corp dashboard.Alerts are simply defined with a handful of fields: <ul><li>Report URL: the URL of the Mode report that this alert can</li><li> fire for.</li><li> </li><li>Alert Title</li><li>Alert Condition: Python expression that is evaluated with</li><li> every row of a report to determine if that row is alert-worthy.</li><li> </li><li>Filter-max Column: name of a column to only take rows where</li><li> the value of that column is maximal. Useful for reports with time-series</li><li> data, usually you only want to alert on rows with the most recent data.</li><li> </li><li>Alert Text Format String: Python format string, used to</li><li> generate the message text in the Slack alert.</li><li> </li><li>Color: sidebar color of the alert message</li><li>Alert Channel: Slack channel to post to</li><li>Enabled</li><li> </li></ul><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F54c463b0ae5740d5af829e3d57c46a96" alt="Report Dashboard"> The combination of the easy-to-use dashboards, and the simple definitions of the alerts, makes creating alerts based on reports very easy to non-technical people—all that’s required to make fully-featured custom alerts is very basic Python knowledge—essentially basic math operations—and SQL if one is creating their own reports from scratch.

Scale relies on Mode Analytics for internal SQL and Python based reporting and analytics, helping power our operations, insights and business decisions.

How to Trigger Alerts With Mode Analytics - Scale

How We Made It Easy for Anyone at Our Company to Trigger Alerts

We have exciting news to share today! Scale raised $18M in Series B funding led by Index Ventures with Accel, Y Combinator, Drew Houston, and Justin Kan joining. Mike Volpi of Index Ventures will be joining our board. Scale has come a long way from the dorms at MIT, and we still have a long way to go. Our mission at Scale is to accelerate the development of AI applications. We are strong believers in the transformational impact of AI and deep learning across every piece of the global economy—from manufacturing and agriculture to autonomous vehicles and security. Deep learning is eating the world in computer vision—models that outperform humans have become standard. It is an important and exciting transition for us—the future will happen very quickly. The bottleneck that becomes apparent to every practitioner is labeled data. The most significant bit to the performance of your AI software is no longer the code you write, but rather the data you are training on. In fact, your code becomes a servant to your ever-growing data corpus—you build hacks to maintain the key invariant of data integrity. The way you improve your software is not to write more code, but to label more data. So when we founded Scale, we devoted ourselves to solving the problem of data labeling. We saw parallels between building web software before cloud computing and AI development today. Before cloud computing, a significant hurdle to building on the internet was building and operating large server farms to power your software. Ultimately, that activation energy prevented many applications from being built. In the same way, a significant hurdle to building machine learning is building the capability to label large datasets. We’re aiming to accomplish exactly what cloud computing did—offer reliable, cost-effective, scalable infrastructure to make it easier to build incredible applications. Scale turned 2 years old last month, and we are well on our way. In autonomous driving, one of the most prominent applications of deep learning today, Scale has become the industry standard for labeling data. We’ve partnered with many industry leaders such as GM Cruise, Lyft, Zoox, and nuTonomy. We’ve labeled more than 200,000 miles of self-driving data (about the distance to the moon). Scale is also used by tech giants like Airbnb and Pinterest, and Fortune 500 companies like P&amp;G and Liberty Mutual. We are proud that our partners have made Scale an integral and trusted part of their pioneering work. Our Series B funding reflects the next stage in our journey, and we are not slowing down. We have a lot left to build, from supporting new applications to creating the best experience around training on labeled datasets. We strive to significantly accelerate the development of AI in every application that makes the future happen faster. To that end, one of our primary goals with the new funding will be growing <a href="https://scale.com/blog/scale-engineering-team" rel="noopener noreferrer" target="_blank">our team</a>. If you want to make a difference in the world of AI, come <a href="https://scale.com/about#jobs" rel="noopener noreferrer" target="_blank">join us</a> on our mission. AI is just taking off. We’re excited to be fueling it.

Scale raised $18M in Series B funding led by Index Ventures with Accel, Y Combinator, Drew Houston, and Justin Kan joining.

Scale raised $18M in Series B with Index Ventures and Accel

Announcing Our Series B with Index Ventures and Accel

Here at Scale, one of the image annotation services we offer is&nbsp;<a href="https://scale.com/3d-cuboids" rel="noopener noreferrer" target="_blank">Cuboid Annotation</a>, which annotates your two-dimensional images with projections of cuboids enclosing objects such as cars, trucks, pedestrians, traffic cones, you name it. With some additional information, we can turn those two-dimensional box annotations into full, three-dimensional boxes, with height, width, depth, rotation, and relative positioning info. <h2>Manual cuboid annotation</h2> With manual cuboid adjustment, scalers simply draw a 2D box representing one side of the cuboid and an additional side of the cuboid: <img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/80ca67fd-00d1-4e55-b7a0-e1d22c826c88" alt="cuboid selection" height="186" width="240">ordinary cuboid This often does not result in a "true" cuboid as it is mathematically imprecise. The front face of a "true" cuboid will likely not be a perfect 90 degree rectangle, especially if it isn't facing the camera head-on. Furthermore, edges of a cuboid parallel to the ground should converge to the horizon, whereas the top/bottom edges of the right side of the above annotation are parallel. In our task response, we provide the image coordinates of the points and edges: <pre class="ql-syntax" spellcheck="false">{
 "edges": [
 {"description": "face-top", "x1": 236, "y1": 308, "x2": 328, "y2": 308, "type": "edge"},
 {"description": "face-right", ...},
 {"description": "face-bottom", ...},
 {"description": "face-left", ...},
 {"description": "side-bottom", ...},
 {"description": "side-top", ...}
 ],
 "vertices": [
 {"description": "face-topleft", "x": 236, "y": 308, "type": "vertex"},
 {"description": "face-bottomleft", ...},
 {"description": "face-topright", ...},
 {"description": "face-bottomright", ...},
 {"description": "side-topcorner", ...},
 {"description": "side-bottomcorner", ...}
 ],
 "label": "car"
}
</pre><h2>Adjusted cuboid</h2> Given the cuboid annotation above, and some additional information (namely, camera parameters and orientation), we automatically generate a more accurate annotation: The front face is no longer perfectly rectangular, but trapezoidal with the left edge slightly smaller, better reflecting the relative orientation of the car to the camera. The top/bottom edges of the right side now converge to a point on the horizon, as they should. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/767c2267-2c47-46e3-91ab-be1281020400/public" alt="adjusted cuboid selection">adjusted cuboid After cuboid adjustment, we add two new fields&nbsp;<code>points_3d</code> and <code>points_2d</code> to the original annotation: <pre class="ql-syntax" spellcheck="false">{
 "edges": {...},
 "vertices": {...},
 "label": "car",
 "points_3d": [
 {"x": 14.903, "y": 4.117, "z": -0.0928},
 ... // 7 more points
 ],
 "points_2d": [
 {"x": 235.39, "y": 308.05},
 ... // 7 more points
 ]
}
</pre>which represent the 3D coordinates of the vertices1 of the cuboid in space (relative to the position of the camera, in meters) and the pixel coordinates of the resulting 2D projection. From these eight points in 3D-space it's easy to infer the position, dimensions, and orientation of the resulting cuboid. With these attributes, one can train models such as <a href="https://arxiv.org/pdf/1611.10010.pdf" rel="noopener noreferrer" target="_blank">Magic Leap's Deep Cuboid Detection</a>, which predicts 3d coordinates of cuboid-like objects from a single image. In this way, one can create a system to identify the locations of cars in the world, using only camera images and annotations generated by Scale. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/e66af6d5-7c8c-4568-a6cd-846a247edd00/public" alt="cuboid annotations">Deep Cuboid Detection Pipeline <h2>Further examples</h2> Initial AnnotationAfter Adjustment <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2dd29cfd-8211-41d1-e0a0-258189a9db00/public"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/f2006f44-0454-466a-f24e-8a9076476c00/public"> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5cca4474-7fce-4132-3e36-fa409ae90200/public"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/c6463b92-2b20-459e-f506-e2aa665efc00/public"> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/50106285-eed1-4ef8-0a8a-f3f263730600/public"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/68665344-da48-47a0-7c9e-eae355031100/public"> Because it's impossible to visually distinguish between one object, and another object twice the size and twice as far away, it's only possible to calculate 3D coordinates up to a scaling factor (i.e. you can multiply the points_3d of the resulting cuboid by an arbitrary value, and it would be just as valid). With additional data, such as depth data from LIDAR or from stereo imaging, one knows the exact distance from the front face of the cuboid and can scale the points accordingly. Alternately, if the elevation of the camera is provided, we scale the resulting cuboids so that the bottom face of each cuboid touches the ground.

Annotating 3d Objects with 2D Data - Scale

Adding a Dimension: Annotating 3d Objects with 2d Data

TLDR: Row stores are fast to write but slow to read. Column stores are fast to read but slow to write. Load data from Mongo into Parquet files for fast querying using AWS Athena. MongoDB is a schema-less NoSQL document store that uses a JSON-like format for each document. Each document in Mongo WiredTiger is stored as a contiguous binary blob, which makes our MongoDB instance a row store. AWS Athena is a SaaS offering by Amazon that queries files on S3 using Presto, a distributed SQL query engine that can query high-performance columnar formats like Parquet and ORC. Here at Scale we use MongoDB backed by WiredTiger as our primary data store. This blog post describes why and how we load data from MongoDB into AWS Athena using Parquet files in order to get fast ad-hoc analytics queries on our data.<h2>Row stores and Column stores</h2> Suppose you want to get the sum of a single column(field) over all the rows(documents) in a table(collection). With a Row Store, without an index, to get a field you need to fetch the document, and to get the same field for all documents you would have to essentially load the entire table from disk. All the columns in your table (even columns that do not appear in your query) get fetched as well, which is slow:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Ff746257aeb8d477d9d6d912f1a328ce7" alt="Data Fields"> You can speed up this query by building a covering index. However, each index you add increases disk usage and slows down inserts. Also, when doing data exploration it can be difficult to predict what indices you need ahead of time. With a Column Store, the query is much more efficient. A column store puts data from the same column together. This would allow us to just retrieve the columns we are filtering / aggregating over when doing queries. This allows us to get good performance on ad-hoc queries, without indices:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fae046f529b8a44e79aa710d117685d41" alt="Column Fields"> Column stores, however, make it very difficult to update a single record - due to the way data is arranged you have to move all of the data to change one record. Also, column stores require us to commit to a schema, to define the columns ahead of time.<h2> </h2><h2> Trading off read and write performance</h2> The slowness of column-store writes is related to the size of the table, because you need to rewrite the whole table to change a single record. By representing each row-store table as multiple column-store tables, we can get the column store to have faster writes at the expense of slowing down its reads. For our implementation, this means we map each Mongo collection to multiple Parquet files (each file actually contains multiple row groups, with each row group being column-major, but we use files as a unit for convenience), and refresh all the documents in a Parquet file whenever any one of them changes. We use smaller file sizes than what would be optimal for reads, in order to make writes faster and reduce replication lag.<h2>Cache Invalidation</h2> When updating the column store from reading the row store, we need to <ol><li>Figure out which records are stale</li><li>Update those records efficiently</li><li> </li></ol> To determine staleness, we need to know the identity of the document and the time of modification. This data is obtained by reading the Mongo oplog, a data structure that represents all inserts/updates/deletes to the database chronologically, available on Mongo replica sets for internal use in replication from primary to secondary nodes within the cluster. For efficiency, we want to ensure that documents to be modified belong in as few files as possible. We achieve this by grouping records by their Mongo <a href="https://docs.mongodb.com/manual/reference/method/ObjectId/" rel="noopener noreferrer" target="_blank">ObjectIDs</a>. We are making use of the fact that, like in <a href="http://wiki.c2.com/?GenerationalGarbageCollection" rel="noopener noreferrer" target="_blank">Generational Garbage Collection</a>, young documents tend to get modified more than old ones. We maintain an index recording the ObjectID ranges of each file, starting a new file whenever the newest bucket is filled with more than N documents.<pre class="ql-syntax" spellcheck="false">class Buckets(object):
 def __init__(self, size_limit):
 self.size_limit = size_limit
 self.buckets = [ObjectId("0" * 24)]
 self.max_bucket = set()

 def find_le(self, x):
 'Find rightmost value less than x'
 i = bisect.bisect_left(self.buckets, x)
 if i:
 return i - 1
 raise ValueError

 def get_bucket(self, oid):
 i = self.find_le(oid)

 # oid is in one of lower buckets
 if i &amp;lt; len(self.buckets) - 1:
 return self.buckets[i], self.buckets[i + 1]

 # oid is in maximum bucket
 self.max_bucket.add(oid)

 if len(self.max_bucket) &amp;lt; self.size_limit:
 return self.buckets[i], max(self.max_bucket)

 # maximum bucket is oversized, start a new bucket
 self.buckets.append(max(self.max_bucket))
 self.max_bucket = set()
 return self.buckets[i], self.buckets[i + 1]
</pre><h2>Implementation Details</h2> <ol><li> </li><li> The scripts that read from mongo and create parquet files are written in</li><li> Python and use the</li><li> <a href="https://pypi.org/project/pyarrow" rel="noopener noreferrer" target="_blank">pyarrow library</a> to write</li><li> Parquet files. The first version implemented a filter-and-append strategy</li><li> for updating Parquet files, which works faster than overwriting the entire</li><li> file. This proved to be a mistake as pyarrow (8.9.0) support for reading is</li><li> much less mature than that for writing. We would occasionally lose data when</li><li> reading rows for rewriting into the new parquet file, and eventually gave up</li><li> and went to an implementation that never read from the Parquet files.</li><li> </li><li> </li><li> Each Parquet file is made of row groups, and there is a size limit for each</li><li> row group. We batch documents by size to avoid this problem.</li><li> </li><li> </li></ol><pre class="ql-syntax" spellcheck="false">def islice_docs(docs, size):
 tot = 0
 for doc in docs:
 tot += len(doc.raw)
 yield doc
 if tot &gt; size * 1024**2:
 return

def batch_docs(docs, size):
 sourceiter = iter(docs)
 while True:
 batchiter = islice_docs(sourceiter, size)
 try:
 yield itertools.chain([next(batchiter)], batchiter)
 except StopIteration:
 return
</pre><ol><li> </li><li> When using pymongo, use RawBSONDocument to defer parsing. In the case of</li><li> multiple processes writing different Parquet files, this means you avoid</li><li> having parsing happen in the serial part of your program. You can</li><li> selectively parse sub-documents using bson.BSON</li><li> </li><li> </li></ol><pre class="ql-syntax" spellcheck="false"> def realize_raw_bson(d):
 if isinstance(d, RawBSONDocument):
 return bson.BSON(d.raw).decode()
 if isinstance(d, list):
 return [realize_raw_bson(v) for v in d]
 return d
</pre><ol><li> </li><li> Use <a href="https://pypi.org/project/atomicwrites/" rel="noopener noreferrer" target="_blank">atomicwrites</a> to</li><li> make writing a new Parquet file all-or-nothing</li><li> </li><li> </li></ol><pre class="ql-syntax" spellcheck="false">@contextmanager
 def atomic_parquet_writer(output, *args):
 os.makedirs(os.path.dirname(output), exist_ok=True)
 with atomic_write(output, mode="wb", overwrite=True) as f:
 with pq.ParquetWriter(f, *args) as writer:
 yield writer
</pre><ol><li> </li><li> On top of extracting explicitly enumerated fields, put the whole json in a</li><li> column by itself for querying fields not in the schema</li><li> </li><li> </li><li> For reading the Parquet files locally using SQL,</li><li> <a href="https://drill.apache.org/" rel="noopener noreferrer" target="_blank">Apache Drill</a> is very convenient -</li><li> just untar and run drill-embedded</li><li> </li><li> </li><li> After updating a parquet file, it has to be copied to S3 (using sync, not</li><li> cp). Using EC2 is essential here because we are constantly copying files</li><li> from the server to S3, and the bandwidth costs would be prohibitive</li><li> otherwise - traffic between EC2 and S3 in the same region is free. After</li><li> copying the file to S3, use AWS Glue to discover the schema from the files -</li><li> call Glue only when the schema changes, AWS charges you each time you call</li><li> it.</li><li> </li><li> </li><li> The oplog contains all the information you need to update the Parquet files.</li><li> We run an instance of mongo on the same machine, and maintain a one-node</li><li> slave as it were by using the applyOps command on oplog entries read from</li><li> the main cluster. Beware, however - Mongo developers might</li><li> <a href="https://jira.mongodb.org/browse/SERVER-33182" rel="noopener noreferrer" target="_blank">remove the docs</a></li><li> for applyOps completely in the future.</li><li> </li><li> </li></ol><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fdf9dcbefbd7f43ce97d5b6780e60ac25" alt="Scale's flow">

How to make MongoDB not suck for analytics - Scale

How to make MongoDB not suck for analytics

We’ve had an incredible group of people join Scale—and we’re still <a href="https://scale.ai/about#jobs" rel="noopener noreferrer" target="_blank">hiring</a>! We wanted to take a moment to call out the amazing engineering team at Scale: <ul><li>Ricky Rauch / <a href="https://twitter.com/rickyrauch" rel="noopener noreferrer" target="_blank">Ricky</a> was the first designer and later Head of Design at <a href="https://auth0.com/" rel="noopener noreferrer" target="_blank">Auth0</a>. He has designed many developer-focused products including <a href="https://github.com/gitlabhq/gitlabhq" rel="noopener noreferrer" target="_blank">GitLab</a>, <a href="https://jwt.io/" rel="noopener noreferrer" target="_blank">JWT.IO</a>, and <a href="http://www.passportjs.org/" rel="noopener noreferrer" target="_blank">Passport.js</a>.</li></ul> <ul><li>Nate Herman / Nate most recently worked on infrastructure engineering at Dropbox. In the past he built an <a href="https://www.wired.com/2012/09/gameception-doom-iphone/" rel="noopener noreferrer" target="_blank">iOS port of Doom</a> and its associated mods, as well as a popular <a href="https://venturebeat.com/2016/08/18/new-pokemon-go-bot-launches-as-niantic-starts-permanently-banning-cheaters/" rel="noopener noreferrer" target="_blank">Pokemon Go</a> automator. He studied computer science at Harvard with a research focus on <a href="http://www.read.seas.harvard.edu/~kohler/pubs/herman16type-aware.pdf" rel="noopener noreferrer" target="_blank">concurrent computing</a>.</li><li> </li><li> </li><li>Steven Hao / Steven graduated from MIT and previously had stints at Jane Street, DE Shaw, and Quora. He received a gold medal at <a href="http://stats.ioinformatics.org/people/3113" rel="noopener noreferrer" target="_blank">IOI</a> and top honors at programming competitions including Codeforces and ACM-ICPC. His personal projects include the collaborative crossword site <a href="https://downforacross.com/" rel="noopener noreferrer" target="_blank">Down For A Cross</a> and a web implementation of <a href="http://set.stevenhao.com/" rel="noopener noreferrer" target="_blank">Set</a>.</li><li> </li><li> </li><li>Leigh Marie Braswell / Leigh Marie has previously worked at Google, Jane Street, and Blend with roles in software engineering, machine learning research, and product management. She was recognized as one of the top 10 competitive female mathematicians in the United States by the Math Prize for Girls Competition. She studied computer science at MIT.</li><li> </li><li> </li><li>Calvin Huang / <a href="https://calvinhuang.me/" rel="noopener noreferrer" target="_blank">Calvin</a> graduated from MIT with a research focus in <a href="https://dreal.github.io/" rel="noopener noreferrer" target="_blank">theorem solving</a> and natural language processing, and with experience at Quora and Google prior to joining Scale. He received two gold medals at <a href="http://ipho2013.dk/ipho2013-results-gold.htm" rel="noopener noreferrer" target="_blank">IPhO</a> as well as the award for Best Physics Experiment, and was a USAJMO winner. He has been known to build robots.</li><li> </li><li> </li><li>Guido Maliandi / Guido previously founded a virtual reality startup for collaborative environments in VR using Unity and three.js. He has led engineering teams at startups for the past nine years and was a major contributor to Microsoft’s <a href="https://prismlibrary.github.io/docs/" rel="noopener noreferrer" target="_blank">Prism library</a>.</li><li> </li><li> </li><li>Zhichun Li / Zhichun has previously worked at Lightspeed China Partners, Facebook, Microsoft and Airbnb with roles in investment and software engineering. She is a producer of <a href="https://mp.weixin.qq.com/s/Kph9YZ1Nvwf4pCt9inT0-A" rel="noopener noreferrer" target="_blank">VC Pulse</a>, a podcast spotlighting venture capitalists in China. Zhichun was the youngest ever admit to the Yale MBA program, and studied computer science at CMU. She has notably been featured on Eminem’s Snapchat story.</li><li> </li><li> </li><li>Rilka / <a href="http://rilka.li/" rel="noopener noreferrer" target="_blank">Rilka</a> previously led design and frontend engineering at a startup and worked as a full-stack engineer at Dropbox. Along with Nate she built a popular <a href="https://venturebeat.com/2016/08/18/new-pokemon-go-bot-launches-as-niantic-starts-permanently-banning-cheaters/" rel="noopener noreferrer" target="_blank">Pokémon Go</a> automator; other past projects include desktop Threes, Facebook Unfriend Finder, and the Quantified Piñata. She studied computer science at Harvard and (briefly) architecture at Parsons.</li><li> </li><li> </li><li>Sam Clearman / Sam received a PhD in mathematics studying <a href="https://vimeo.com/123640110" rel="noopener noreferrer" target="_blank">combinatorial interpretations of quantum immanants</a>. He was previously an engineer at Google working on Google Play Services. He also built a variant of Tetris called <a href="http://www.samclearman.com/2etris/" rel="noopener noreferrer" target="_blank">2etris</a>.</li><li> </li><li> </li><li>Chiao-Lun Cheng / Chiao received a PhD in computational chemistry from MIT, and has a diverse background in management consulting, quantitative trading, backend engineering, and data science. He has built models for supermarket revenue prediction, bookmaking sports bets, and PCB defect detection, and is intimately familiar with bugs in the Interactive Brokers API.</li><li> </li><li> </li><li>Akshat Bubna / Akshat was the first competitor from India to receive a gold medal at the <a href="http://stats.ioinformatics.org/people/3063" rel="noopener noreferrer" target="_blank">IOI</a>. He previously has worked at DE Shaw and database and fintech startups, and studied computer science and mathematics at MIT.</li><li> </li></ul><h2>Contractors</h2> <ul><li> </li><li>Albert Sheu / Albert was the second engineer at Quora and built a Python framework to render webpages on multiple cores, which sped up Quora by <a href="https://blog.quora.com/Improving-Site-Speed" rel="noopener noreferrer" target="_blank">up to 10x</a>. He also was the Head of Engineering at Patreon and an early engineer at both Facebook and Twitter.</li><li> </li></ul>

Blog