<p>Aerin is an engineering manager at Scale AI, leading a Catalog ML team. Below are Aerin's stories about what she does and why she loves working at Scale.</p>
<h2>What brought you to Scale?</h2>
<p>Before I joined Scale, I worked on a project called NL2SQL which stands for "Natural Language to Structured Query Language" at Microsoft. NL2SQL is a translation problem that automatically converts a user's spoken-language query into SQL.</p>
<p>Just a few months after my team had started working on it, we were beating the SOTA (state-of-the-art) performance on research benchmarks. We achieved this by evaluating different ML model architectures and improving them. In other words, we somewhat solved the NL2SQL problem in research after only a few months of work.</p>
<p>However, in production, we weren&rsquo;t breaking the state of the art. The model still made a few mistakes on the user's input. And it became clear to me that changing the model architectures would only get us so far, and that the most crucial thing we could do to improve the performance of NL2SQL was to improve the training data. This was done by generating edge cases synthetically and therefore increasing the training data coverage.</p>
<p>So, we started synthetically generating training data (NL-SQL pairs) by following syntax requirements, snowflake schema, etc. However, the data that we have artificially created couldn't possibly cover every possible scenario or edge case. Because the true edge cases are ones that you can't predict or plan for, like the questions that people have typed into our search query bar.&nbsp;Another way to increase the coverage of the training data was to label the user inputs and feed them back to the training data. Taking care of that lifecycle, also called an auto-labeling pipeline, was quite an operation. Even though I was given a lot of resources to manage the auto labeling pipeline, I couldn't undertake any other engineering work because it required so much overhead.</p>
<p>So I started looking for the best third-party training data provider in the market, and that's how I met Scale, and the rest is history.</p>
<h2>What do you enjoy most about working at Scale?</h2>
<p>I like that I get to work on problems that are important and challenging enough to customers that they are willing to pay to have them solved. This means that the problems are real.</p>
<p>I also like that Scale's work is always about data, which is my favorite subject. For every business, I believe data is the most valuable and private asset. Say if you're developing a machine learning model, the one and only input that determines how well your model will perform is your data. The fact that customers trust us with their data and allow us to manage it energizes me, and I feel a great deal of commitment and responsibility. Because the data we provide with them is being used as a bloodline for their business, this relationship just can't work without trust. Having a close relationship with customers is also a rewarding part of my job.</p>
<p>Another highlight of my time here at Scale is the everyday connection with my team. When the engineering team works well, it feels like a well-oiled machine, and you will notice tangible results from that. We've done some seemingly impossible jobs together, and it feels amazing to see the whole team pulling together to complete a challenging project. When working on projects, you learn a great deal by openly debating and choosing the best technical solution with your team. If you're an engineer, I believe that seeing and interacting with other engineers is one of the best ways to learn and grow. That's also something I enjoy at Scale.</p>
<h2>What makes working for a startup unique from other jobs?</h2>
<p>I've worked at both a big tech company and a startup, and they are indeed quite different, and each one is unique in its own way.</p>
<p>At a startup, leading a project feels like driving a sports car. You're close to the ground and can feel the movements you make. You can turn quickly, and you can observe fast motion while you are steering. You can see the results of your team's work fairly quickly and clearly. For example, you complete a pilot project successfully, and the customer signs a large contract. There is also a good chance of crashing - for example, you fail to deliver a project and lose a customer. At a startup, you will see the results of what your team does right away.</p>
<p>At a big company, running a project is more like flying a spaceship (I can only imagine this). You&rsquo;re farther from the ground, it's harder to make quick decisions because you are not the only stakeholder, and the course has been set long in advance. There are also very well-built manuals and processes to guide you along the way.</p>
<p>So far, I've enjoyed seeing the immediate, direct effects of my team's work at Scale.&nbsp;</p>
<h2>What's your favorite part about being an engineering manager at Scale?</h2>
<p>One of the things I like about being an engineering manager is that I get to work on large-scale projects with a strong, cohesive team, which results in a scalable solution and a greater overall team output. Apart from the collaborative and multiplier nature of the job, I also enjoy problem solving at Scale.</p>
<p>People think that a software engineer's job is to write code, but I don't think that's a complete picture of an engineer. A more accurate job description would be "a problem solver", since our main goal is to build things that help people solve problems. You won't be able to solve problems if you blindly write code, especially from the wrong angle.</p>
<p>I believe the most important thing to do in order to solve problems is to accurately diagnose them. So the ability to diagnose problems accurately is, in my opinion, the most important skill for an engineer to have. If your diagnosis is wrong, you will rarely be effective in solving actual problems. And you can only make the right diagnosis if you have a full, concrete understanding of the problem, the data, and the use cases.&nbsp;</p>
<p>While surgeons can rely on other professionals like physicians, x-ray technicians, to aid with diagnosis, ML/SW engineers have to diagnose problems on their own. We have to carefully examine the available data, debug the code thoroughly, then choose the best solution and also implement. The ML/SWE profession comes with a significant amount of both power and responsibility.</p>
<p>If you are someone who enjoys doing things like this (proper diagnosis and implementation), you will do well in the field of engineering.</p>
<h2>Tell me about interesting projects that you ran at Scale?</h2>
<p>Every project at Scale is interesting and unique in its own way, as each project has different data modality (3D Lidar, 2D imagery, NLP, tabular, etc.) I enjoy that I get to work on diverse problems and apply the peculiarities of each data modality to ML model development. For example, 3D data often comes with the new axes - yaw, pitch, roll, unlike 2D data and the sensor's effect on the LIDAR data is significant. NLP also has its own quirks, such as different types of tokenizers, vector representations, etc.</p>
<p>These days, my team is working on multi-modal (Language + Vision) problems. For example, given an image of a model wearing clothes, can we describe what she&rsquo;s wearing in detail (e.g. short sleeve, polka-dots, long boots, etc.) in natural language? To this end, one popular research/exploratory approach is to train a multi-task (e.g., Visual Question Answering, Image Captioning, Text-to-Image Generation, Visual Grounding, etc.) model and use it for attribute prediction.</p>
<p>Although I can't talk about the details of the projects I work on for clients, I do make an intentional effort to make my team's work more universally applicable, scientifically relevant, and therefore publishable. The attribute prediction work (link) that was just accepted at CVPR 2022 is a great example. One of the most common requests we get from clients is to assign visual characteristics such as color, shape, or motion to detected objects. But there aren't many researchers working on this topic publicly (though there are a lot of practitioners working on it in industry), and there aren't many benchmark datasets for attribute prediction. So we did in-depth research on this topic, presented our findings at CVPR, and received overwhelmingly positive feedback from the ML community.</p>
<p>The Fitzpatrick 17k project with MIT is also a good example. The Fitzpatrick scale is a 6-point scale that tells how dark your skin tone is. And this project&rsquo;s motives are quite fascinating. Because medical imaging doesn't have much data on darker skin, ML models trained on these datasets have trouble recognizing dermatological illnesses on darker skin.&nbsp;We wanted to bring attention to this problem and encourage more inclusion of images with darker skin tones in datasets. So we annotated 17k clinical images in DermaAmin and Atlas Dermatologico, the two major dermatology atlases, using the Fitzpatrick Scale.&nbsp;</p>
<p>We trained a few models on this new dataset and then looked at how well they work on images with different skin tones. And yes, adding more diverse data to the training set was an effective way to reduce bias.&nbsp;</p>
<p>Because of this work, researchers and practitioners&nbsp;are now a little more likely to take into account and examine biases in training datasets before developing ML models in the healthcare sector. Ultimately, we want to make sure that ML models trained on imbalanced datasets don't make healthcare inequalities worse by accident. With this study, we received an honorable mention award at the CVPR skin imaging workshop.</p>
<p>&hellip;&hellip;..</p>
<p>If you are interested in any of the topics that I have said above or in working to solve intriguing ML/SWE challenges, email your resume at <a href="mailto:aerin.kim@scale.com" target="_blank" rel="noopener noreferrer">aerin.kim@scale.com</a>!</p>

Read Aerin's stories about what she does and why she loves working at Scale

Aerin is an engineering manager at Scale AI, leading a Catalog ML team

How Nick Beighton transformed ASOS into a 4 billion dollar business by investing in product catalog data

<p>We've had an incredible group of people join Scale—and we’re still <a href="https://scale.com/careers?office=All&amp;department=All#jobs" rel="noopener noreferrer" target="_blank">hiring</a>! Meet the engineers who are accelerating the development of AI applications:</p><h3><br></h3><h3>Albert Sheu</h3><p>Albert is a fifteen-year veteran of the tech and startup industry, including having worked as the second engineer at Quora and the Head of Engineering at Patreon. His experience runs up and down the stack, working on distributed systems, developer experience, WebGL interfaces, and all areas in the frontend and backend. At Scale he developed the early 3D labeling pipeline, and has also worked on Scale’s Nucleus and Synthetic Data products.</p><p><br></p><h3>Jack Wu</h3><p>Jack Wu, previously was a manager at Niantic where he built the Platform Infrastructure team from scratch and led the team to rebuild the Niantic million QPS, planet scale Augmented Reality platform on an alternate Cloud provider. Prior to transitioning to management, Jack was an early engineer at Uber and LinkedIn where he spent most of time building large volume data storage, data analytics and data security systems and saw both companies grow more than 10X during his tenure.</p><h3><br></h3><h3>Yuting Duan</h3><p>Yuting worked at Google as a Staff Software Engineer and Engineering Manager and studied at CMU. She worked on some of the most large-scale consumer-facing products: Google Search and Google Maps and founded 2 full-stack teams in Google to help users find food wherever they are, whenever they are.</p><p><br></p><h3>Shah Xia</h3><p>Shah has previously worked at Bytedance where she was the Chief of Staff to the CEO. She was part of the early team to incubate the TikTok app, and later directed its global expansion strategy. She is receiving her MBA from Wharton.</p><p><br></p><h3>Sam Clearman</h3><p>Sam Clearman received his PhD in Mathematics at Lehigh for his work in algebraic combinatorics.&nbsp;He worked at Google using statistical methods to improve Android battery life before joining Scale in 2018.&nbsp;At Scale, he has worked on pretty much everything, but he has always returned to marketplace management and incentive design.&nbsp;In his spare time, he flies planes (sometimes upside down), and enjoys making games such as 2etris [<a href="https://2etris.com/" rel="noopener noreferrer" target="_blank">https://2etris.com/</a>], Down for a Block [<a href="http://d4ab.com/" rel="noopener noreferrer" target="_blank">http://d4ab.com/</a>], and Up for a Camel [<a href="https://upforacamel.com/" rel="noopener noreferrer" target="_blank">https://upforacamel.com/</a>].&nbsp;</p><p><br></p><h3>Sasha Harrison</h3><p>Sasha Harrison received a Bachelor's Degree in Mathematical and Computational Sciences, and a Master's Degree in Computer Science from Stanford University, where her concentration was in Deep Learning. As a student, she conducted research projects related to algorithmic fairness in computer vision and NLP, and was an intern at Google and Microsoft. At Scale, Sasha has worked across new products and ML, in particular, helping to develop scalable batch inference infrastructure as well as deep learning models for application to the content understanding data annotation vertical.</p><p><br></p><h3>Calvin Huang</h3><p>Calvin Huang met Alexandr Wang at the US Physics Camp just before starting to study electrical engineering and computer science at MIT.&nbsp;While at MIT, Calvin did research in SMT solving and natural language processing. After graduation, Alex recruited Calvin to join Scale as an early engineer, where he built out a large portion of the infrastructure stack and developer tooling, and helped the core systems scale as Scale did.</p><p><br></p><h3>Vivek Muppalla</h3><p>Vivek is an industry veteran who has worked in and lead intraprenual efforts across startups and large companies. His experience is leading full stack engineering and product teams. Most recently he was an early member of Unity’s AI &amp; simulation team, and helped Unity become a leader in the Synthetic data and Robotics simulation space. At Scale, he is focused on building our Simulation based products.</p>

Meet Scale's incredible engineering talent 

Scale's Talented Engineering Team

<p>As ML applications grow in size and complexity, testing is essential to ensure integrity and reliability of the application at hand. Given the non-deterministic nature of machine learning (ML) models, testing and continuous evaluation of ML systems is even more important.&nbsp;</p>
<p>That&rsquo;s why today, we&rsquo;re thrilled to announce Early Access to <a href="https://scale.com/nucleus" target="_blank" rel="noopener">Scale Validate</a>, our testing solution for ML models. The predictive performance of ML components is essential to the overall success and adoption of whole products. Progressively developing a high-performing ML-based system is not trivial, because it typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires continuous iteration and experimentation</a>. During these iterations, engineers need to identify new edge cases and improve model performance on those. With every new model iteration, ML engineers want to be sure that they aren&rsquo;t introducing any performance regressions on mission-critical subsets of data.&nbsp;</p>
<p><strong>Using Scale Validate, you can now define data-centric tests for your ML models to avoid performance regressions during development. This step helps ensure that only the best ML models make it into production.</strong></p>
<p><a href="https://scaleai.wistia.com/medias/ks6a02aqwq" target="_blank" rel="noopener noreferrer">Video</a></p>
<p>For ML-based products, model performance should always directly correlate with positive business outcome.&nbsp;The performance of ML components is often crucial to the success and adoption of whole products. Getting to a high-performing ML-based system is not trivial but typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires many iterations</a> and will almost never be done. During these iterations, engineers need to identify new edge cases and improve the model performance on those. With every new model iteration, ML engineers want to be sure that no performance regressions on mission-critical subsets of data are introduced.&nbsp;</p>
<p>While ML model testing is key to fast and robust model iteration, there are a few common issues that we identified with existing testing and evaluation pipelines for ML models:</p>
<h4><strong><em>Aggregate evaluation metrics miss critical regressions</em></strong></h4>
<p>A common way of validating ML models is to compute aggregate scores and metrics (e.g. mAP, precision, recall, IoU, &hellip;) across fixed test/validation datasets (commonly understood as model evaluation). Evaluation with aggregate metrics is necessary but not sufficient for robust models because they often miss failure modes that are underrepresented due to data biases. Users won&rsquo;t be able to track regressions with explicit checks on such failure modes that were previously addressed but are failing now.</p>
<h4><strong><em>Coverage for ML model testing is not well defined</em></strong></h4>
<p>While we can analyze covered lines of code in software testing, there&rsquo;s no such number which informs about the coverage of tests for ML-models. Lines of code still need to be covered, but more importantly, tests need to cover the operational domain of the model input data. The high dimensionality of typical inputs (e.g. an image has more than a million pixels with 3 channels) does not allow for realistic or efficient input-fuzzing approaches.&nbsp;</p>
<p><strong><em>Testing pipelines are static and don&rsquo;t evolve with the models</em></strong></p>
<p>Building ML models is an iterative process. So is testing ML models. Not adapting testing and validation datasets results in vanity metrics which in turn oftentimes overestimate the confidence in the models. Continuously evolving test definitions and adding the most tricky examples is crucial to continuously improve the test coverage in the operational domain. This is the only certain path to confidently deploy robust models to production systems.</p>
<p>Accordingly, we built Scale Validate with the following capabilities:</p>
<h3>Scenario Tests</h3>
<p>Setting up unit and integration tests is the go-to practice to ensure correct behavior of codebases. For ML systems, unit tests are designed to test the correctness of individual model components, which is necessary for a reliable model performance. Scenario tests are designed to test the actual model performance and decision-making capabilities of ML models on mission-critical scenarios in the operational domain. Scale Validate allows users to easily define scenario tests on subsets of data (<a href="https://dashboard.scale.com/nucleus/ut_camxgk2yxqgg05rzc4bg" target="_blank" rel="noopener noreferrer">example</a>), using off-the-shelf or custom evaluation metrics. New models are automatically tested against defined scenario tests. Well-defined scenario tests allow ML engineers to catch regressions <em>before</em> deploying a model to production. When a test fails, users immediately see in which scenario the required performance is not achieved anymore (<a href="https://dashboard.scale.com/nucleus/prj_cabt0xkhtxbg048dvqn0?spoof=select%40scaleapi.com&amp;val_ev_gsp=2165&amp;from_unit_test_id=ut_camxgk2yxqgg05rzc4bg&amp;model_page_display=Evaluations&amp;val_model_gsp=1154" target="_blank" rel="noopener noreferrer">example</a>).</p>
<p><img src="https://lh6.googleusercontent.com/lyc0KYEFc_1GRv-k5m7BPrtijOlmb4ef2xg6J_oVdWJFimn3rfwTjMkpaYs3aUu5PUJI5VjPExMjvC3tuK3V6LLEuAXPlZDRwoT1wfXaTfmFBeZxuq8ov_3OLitltffWIaU_-Q7RJ2y1XbZefMfzkXI"></p>
<h3>Model Comparison</h3>
<p>Model performance rarely boils down to a single metric. With Scale Validate, it&rsquo;s trivial to compare the performance of new models against baseline models (<a href="https://dashboard.scale.com/nucleus/analyze?analyze_compare_model_id=run_cabp73ckv9wg078pk440&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;q=predictions.count+%3E+0" target="_blank" rel="noopener noreferrer">example</a>). You can use either our standard performance metric tracking across scenario tests or our side-by-side model comparison. All charts are fully interactive and users can start by informing themselves from higher-level metrics, then graduating into learning from item-based analysis.</p>
<p><img src="https://lh5.googleusercontent.com/Ny6n_eMIjJ1kMGRDpWyp47In_FjhErjXbVUS1ou5VvweGqSo_oDjyd4ORb07DmiCrSBPIbkhxQz-v840Nlaj8DmKzFny66BCVsBh9b1z0gMw23Wgq86jkpasU8QdBzp90S9ER5BQBKAIuPAhodUT8e8"></p>
<h3>Slice Discovery</h3>
<p>Scale Validate enables users to automatically identify coherent and underperforming slices of data that do not yet have test coverage. Users can be sure they are always testing against a representative test set.</p>
<p><img src="https://lh3.googleusercontent.com/tsjA4raZ7xVSuMM9FZR-gLVSn-pX_r99a1r-2hPCTyub75QNG5J6SSRXf2J5c7WiQtWZIHyEgkS78j-ST7u2qooBIDEXC7xZcGKbo6ojWQmdmhBkkLiFPkF7YS7gDw4j5-uvbhjsUXC67OuuxT451fU"></p>
<p>We built Validate as a developer-friendly solution both for API-first and UI-focused users. The Validate interfaces and endpoints make it as easy as possible to get started and integrated into existing pipelines. Today, Scale Validate is used for the testing pipelines of leading ML teams in areas including autonomous driving, surveillance, and industrial robotics.</p>
<p>As Pickle Robot&rsquo;s CTO, Ariana Eisenstein, explains:</p>
<p><img src="https://lh6.googleusercontent.com/gb9ujMPiBfQOhkiz36npqFR8kY44KymYOC79ac0AQaL4UAK9H2Nm7w3O5bLE7V0xF6w3tOe2E1eRrl57hDs_FMOaZCm9HbzFutL6UmX_s2mPuJ_psw-Aoub5rNo9jynrD_q0yxbtDG8o-QFEwHEkHxQ"></p>
<p>Getting started with Validate only takes a few steps!&nbsp;</p>
<ol>
<li>Sign up / log into the <a href="https://dashboard.scale.com/corp/login?redirect_url=%2F&amp;clear=true" target="_blank" rel="noopener noreferrer">Scale dashboard</a></li>
<li><a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">Explore Validate with public datasets</a></li>
<li><a href="https://nucleus.scale.com/docs/datasets-in-nucleus#uploading-your-data" target="_blank" rel="noopener noreferrer">Upload your data</a></li>
<li>Upload <a href="https://nucleus.scale.com/docs/model-predictions-in-nucleus" target="_blank" rel="noopener noreferrer">models</a> and <a href="https://nucleus.scale.com/docs/ground-truth-annotations-in-nucleus" target="_blank" rel="noopener noreferrer">annotations</a></li>
<li>Compare your models in our <a href="https://dashboard.scale.com/nucleus/analyze" target="_blank" rel="noopener noreferrer">dashboard</a></li>
<li>Select interesting slices of data and <a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">define tests</a></li>
</ol>
<p>Without a detailed model evaluation and testing pipeline, you&rsquo;ll never know which performance points you&rsquo;re leaving on the table and which performance regressions might sneak into your models. Instead, ensure that your model quality only increases with every model and dataset iteration. Test-driven modeling will ensure that you train a production-grade model faster! If you&rsquo;d like more information about Scale Validate, reach out to us for a detailed demo and discussion <a href="https://scale.com/validate" target="_blank" rel="noopener noreferrer">here</a>.</p>

Introducing Scale Validate for Model Testing

<p>If code quality is what sets apart good software systems from bad ones, data quality is what distinguishes great AI systems from the ones that will simply never make it to production. While most of the research in machine learning (ML) continues to focus on models, industrial players building AI systems are focusing more and more on the data. The number of companies adopting a <a href="https://datacentricai.org/" rel="noopener noreferrer" target="_blank">data-centric approach to AI</a> development (understood as systematically engineering datasets used for AI) is growing rapidly across industries and levels of company maturity. According to former Tesla Autopilot lead Andrej Karpathy, engineers at Tesla lose sleep over the dual challenges of structuring their datasets effectively and handling the edge cases that might come up.</p><p class="ql-align-center"><img src="https://lh4.googleusercontent.com/bGi4IgNBP_6iLqhMvrAwXJFXRzwRHiQt5iMhU1jvUgHC44K-eJFb5uO_tPueneP6pZu9fkxsWnWUsMhITkK6I2acFC52sC3K_Kzhhk0ogXqNPFHfZD5V7HDx1bQhliZIX-UYR_KKBZjNlkTlswb28I0"></p><p class="ql-align-center"><em>Graphic adapted from Andrej Karpathy’s slide deck [Train AI 2018]</em></p><p>But why is using the right data so central for building out high-performing ML models? The answer lies in unequal data distribution and low-quality ground truth (annotations).</p><p class="ql-align-center"><a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Try Scale Nucleus</a></p><h3>Not all data is created equal</h3><p>The core hypothesis behind data-centric AI is that <a href="https://scale.com/blog/improve-data-annotation" rel="noopener noreferrer" target="_blank">you can achieve better ML model performance by training your model on higher-quality data</a>. In an ideal world, your model performs equally well in all kinds of situations. For example, in autonomous driving, you would want a model detecting pedestrians to work equally well, irrespective of the weather, visual conditions, how fast the pedestrian is moving, how occluded they are, <em>et cetera</em>. Most likely however, your model will perform much worse on cases that are more rare—for example, a baby stroller swiftly emerging from behind a parked car in inclement weather.&nbsp;</p><p>The point of failure here is that the model has been trained on data that was recorded during regular traffic conditions. As a result, the representation of these rare scenarios (as a portion of the entire training dataset) is much lower compared to common scenarios. Below is another example of two highway scenarios, whereas lane detection will be significantly more difficult in the right hand picture compared to the left.</p><p class="ql-align-center"><img src="https://lh5.googleusercontent.com/Nr-B0CIcVGPiYOS9BRizyV96w9k_tLKcdLpaDpYzoZeTfDB2LOY5F5--BKhH70lyDC9ZBvqxoL2ks2x1C_qIfoO6cBaScI-SkyFTGUCv3GU5NjkT8GpsF788mQpm-bl1yqnVMabnW8umPNOajNgySlE"></p><p>What can we do about this? We need to increase the prevalence of these rare cases in our labeled training data.</p><p class="ql-align-center"><img src="https://lh4.googleusercontent.com/8VXzOph0ZPD3peHbzOCCOgX1Ecx3TbroIKlFlD0-il7MTqFkLVHz7HBwRaYwAINnOMzTZnussN0dS-ivxzk4Flb5VX37a-uAvlXhMuuVKXHBpvakXAkUFkMjQuW_lmxsdpwynICGXNNz_pVCPtR6bX0"></p><p class="ql-align-center"><em>Accuracy improvements with data selection via active learning versus random sampling given a fixed labeling budget.</em></p><p class="ql-align-center">Source: <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Emam et. al., 2021. Active Learning at the ImageNet Scale</a></p><p>In ML research, this approach is often referred to as active learning. Many papers (e.g. <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Active Learning at ImageNet Scale</a>) have demonstrated the effectiveness of this approach, showing significant performance improvements by picking more difficult (most often underrepresented) training data instead of random sampling. At Scale AI, this is also what we observe with our customers, especially those with advanced ML teams and models deployed in production. Companies that are the best at picking the right data for labeling—and eventually training—have the highest performing models. Unfortunately, identifying the highest impact data is difficult, and most ML practitioners perform this task manually, even when wrangling very large datasets.</p><h3>Labeling errors are breaking your model</h3><p>To make things worse, almost all datasets include labeling errors. Labeling errors are more likely to occur on edge cases. This makes sense, because images that have been seen 1,000 times are easier to label correctly compared to an image seen only once. Labeling errors are also most destructive for your model performance if they happen on edge cases, because you have a limited amount of these samples in the first place. So if the labels on these difficult samples are wrong, model performance will never improve. Instead, it’s more likely to quickly regress.</p><p class="ql-align-center"><img src="https://lh4.googleusercontent.com/rFWziHErcB0S1tlTzQQDw2OWyhFgaCHQJumi-rYeMg0Wtmql4_Xr31QXo8bZrBo_xt_Nhk5_XflX7BEW4cOaJmoqS0bVcrPviFCzeKQC8wEAQbytWbb-FsJghbtm0tnqREVjq_NJdoplyM3u6_UEFxs"></p><p class="ql-align-center"><em>Example of a typical l labeling error on the public Berkeley Deep Drive dataset: class confusion between car (the annotation)&nbsp;and bus (the prediction).</em>&nbsp;</p><p>As a result, finding and cleaning bad labels (generally, but especially for rare cases) is probably one of the most impactful activities you can do to meaningfully improve model performance. Many teams do this already. Unfortunately, though, doing this consistently, especially on large datasets while keeping track of issues and measuring progress, is often a very manual and isolated process. Many times, teams will use Jupyter notebooks to find bad labels and keep track of them in spreadsheets.</p><p>Doing things manually will only work for so long, and it will not get your model into production anytime soon. To address this we have built Scale Nucleus, a data management tool that helps to decide what data to label and make sure that these labels are consistently of high quality.</p><p>In the rest of this article, we’ll walk through common strategies to mitigate the problems outlined above, starting with unequal data distribution.</p><h3>Use similarity search to get from one failure case to many</h3><p>One of the most common ways to build good training data is to look at a single sample that is particularly difficult for the model (e.g. based on pure luck, a field report, domain knowledge or other data points) and try to find more of the same.</p><p class="ql-align-center"><img src="https://lh3.googleusercontent.com/JKZZC8hjToji6ebsTT9Tm6iAk-nY3gX-BPs4j-Ojc76bp_J3BXcT-kU1vZJ9njE2f0KYt3opfz9n8cQW2bf5_DMRB40XtXkvBfRbkw7P6Y-tVpKTCU4mz7-Z9jFFh5soMJ6t79KOoyxZDMzojbi_FRk"></p><p>In Nucleus, you can do this by selecting a few images of interest in your dataset and running a similarity search. For example, you might know that perception performance is bad on blurry images. In this case, you’d select a single blurry image and run a similarity search.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F3c1101b17e1d4d95a3030dec3104f835"></p><p>However, a single similarity search might yield mixed results; not everything is equally relevant. Furthermore, you might want to codify the issue you were looking for into a “tag” and consistently find all images that have this issue. In our example, you might want to make sure every image in your dataset can be identified as blurry or not blurry going forward.</p><p>One way to address this is using a Nucleus Autotag. Autotag is a way of teaching Nucleus about a new concept, by interactively selecting more and more specific examples of a case, for example blurry images, in a few refinement steps. In just a few minutes, the platform will “learn” a concept like blurriness and enable you to search your data using this attribute going forward. In the background, we are training a binary classifier, which assigns and updates a similarity score for the concept you are searching for across your whole dataset. You can learn <a href="https://nucleus.scale.com/docs/autotag" rel="noopener noreferrer" target="_blank">more about Autotag here</a>.</p><p><a href="https://scaleai.wistia.com/medias/gd6yfivwwk" rel="noopener noreferrer" target="_blank">Here</a></p><h3>Find rare edge cases with natural language search</h3><p>Many times, an ML engineer might know (e.g. based on domain expertise) what kind of edge case data is difficult for their model, but does not immediately have an example of this at hand. Edge cases are often also environment conditions or situations, for which you have neither annotations and nor does your model predict them—yet. Examples might be specific lighting conditions, occlusions, or novel objects types.&nbsp;</p><p class="ql-align-center"><img src="https://lh6.googleusercontent.com/ZVoRKE_yYTYVkKMTLeXN1vuh-3ExukAYnhTIPL9vtW55Bee0ePoKU5tycAWXJwytidezHuu_f5jvbInP5-P2GLYlqUPmTS-WM0ocfqEVwfJr-G1lNSmUIIHtPhwYnu3f4ke3tGXqoutxCCkXzBOhPmw"></p><p>To start, you might run a Natural Language Query to find these examples. For example, you might want to find <a href="https://dashboard.scale.com/nucleus/ds_bwk18y3e696g05rk8gyg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+pizza+box" rel="noopener noreferrer" target="_blank">all frames of pizza boxes</a> in the famous ImageNet dataset. It is important to note that this style of query works on data without any annotations, model predictions or metadata because the samples you are trying to find are edge cases which are not yet detected by your computer vision model.</p><p>One way of solving the “no labels challenge” is to use precomputed embeddings that are matched against a natural language query, e.g. using OpenAI’s <a href="https://github.com/openai/CLIP" rel="noopener noreferrer" target="_blank">CLIP</a> model. Nucleus generates CLIP embeddings automatically for all images in order to support natural language search.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">Natural Language Search</a></p><p>Once you’ve built a more balanced dataset and enriched it with annotations, now you’re ready to ensure these labels are consistently of high quality, especially for edge cases.</p><h3>Review computer vision labels</h3><p>Now that you have a set of image annotations, you’ll want to verify that they are of high quality before using them as training data, especially for rare edge cases. This can either be done image by image, or in batches as long as the data annotations can be easily inspected and compared against the actual image content.</p><p><img src="https://lh3.googleusercontent.com/AdVeuzXqgQzQKvjjMaP4j2Dco5iGD7CLrs2FXOriHzr0I48c4J732dkeZ6Ynx_TAPlculOjwyWnWKB0HinRuzaXG--Ays5RULi1ptcMF8DmPqxKZodGnFc-gDnx98xNsZjfyzFM995-yG2wMUZkPcWM"></p><p><img src="https://lh4.googleusercontent.com/GYD7xBOcas6YKhtCKoHZWwxib2KGMRJXXRN-X_hwGzek_UO37C_gQiatduhSZKO5sKXPd09li3pDYA64kMqeN3viU8IjhYCtMq5AkNsbK-xOg7AnN5gd9KSdQqy0XF5EhGtoN_yCrxt8gOl-QYRtUwA"></p><p>To improve label quality in the next iteration of labeling, it makes sense to mark each individual frame and bounding box as accepted or rejected and later query against these attributes to make sure all items are reviewed. If you encounter a particular issue, such as a badly fitted bounding box, a good option is to add the relevant images to a “slice”, which is a curated subset of data to share with colleagues or to return to at a later point.</p><p class="ql-align-center"><img src="https://lh4.googleusercontent.com/UtXwA46VnC7onh-V4MrgPgF8cFHJZ3wxDeL4NJEb8PGoX8atpZsRSRMR9uIN29g5W4uVrjGxHve6mZHk734ZBAQpvY86X-nE9JaKN832GKyY3d4_oUubJR4L-DkVF5997tBqog57l1aNYKDfPTnAu7g"></p><h3>Find mistakes in computer vision labels</h3><p>Instead of going through all your data manually, it is often advisable to compare the labels you are reviewing against the predictions of your most recent computer vision model. There are several so-called model-assisted methods to find and clean labels such as sort-by-loss, sort-by-confidence where model and ground truth disagree, or confident learning (<a href="https://jair.org/index.php/jair/article/view/12125" rel="noopener noreferrer" target="_blank">paper</a>). It is in fact often surprising how effective methods such as sort-by-confidence are to improve label quality. ML practitioners do this by computing common error metrics such as the aggregated IOU, a confusion matrix, or a precision-recall (PR) curve (e.g. in a Jupyter notebook) and then inspecting the samples where the prediction of the model and the data annotation are least aligned.</p><p class="ql-align-center"><img src="https://lh3.googleusercontent.com/me0p6swzqwP1_AN4g8O4BbGBuKNb0LqJLgqr240I39VTN2a-jaHLYSC-VUn5fJAFkhf6z9ClYSg9OKZs5TC-UEJzEA1tzg5IwMoiH-2Auf0ECZOJhUwxT-uIIhgGh0TNaWfkXHW0KsI2A6hrv7D8vmo"></p><p>However, using a GUI this process can be 10x faster, because error metrics are pre-computed automatically and can be directly explored with interactive charts. Clicking into any graph automatically queries and surfaces all the samples associated with that data point. For example, clicking into the cell for “bus versus truck” confusions in the matrix displayed below queries your dataset for all samples where the data label is a truck but the computer vision model predicts a bus.</p><p><img src="https://lh4.googleusercontent.com/9GiOAPH8IoxI6QiMfYa7Ia7rOPcvwDTZuuMaDkPTG7-176VwjOM5cjZmLUGNXEBvwtpCclB3Ryp3e8vQV6BHMpEagh14TbEc_L90i7vuKb7KX04Xkvxt_Pjap7Ta7LShMBKcRzYfVDG35ecCdaEdjJE"></p><p><img src="https://lh4.googleusercontent.com/XFAtHxplLOV_1YHFQJv6Bb80wU2qce2hu-20b-D-BzkIIJ0n8xl6iNAgdxMuUse3ckJdEcbGUEEMzC1qhyNAdzls6lFQiU8P6JkPLZd1KxuRY_Iwuy6zS_ZQZMvVvtfMUZqeQX-6RI-A6-sbYMWVXrE"></p><p>From just a single glance, we can see that these are buses that are incorrectly labeled as trucks. These image annotations need to be corrected to avoid bad model performance for both bus and truck detections.</p><h3>Find missing labels</h3><p>Finally, it is critical to ensure that no labels are missing in your annotated image data. A missing label can lead to a performance regression of your model, especially if the object with the missing label is easily detected by your model.</p><p><img src="https://lh4.googleusercontent.com/IlPV_qOBq4bmtcTAEP119G257XAKc3W5titad6vm2VF0rZ25lIGQhALm2PcLwlB1QUS8K3bTKf17Lcqf24zed-f-mPhLtPEJqk9V9m-eF6Bnz6D7ueyoeSqs10ra1YO_PHqXwFmtf9LO30pJbSQul2Y"></p><p>In the above example, all cars are annotated in the image, but the single white vehicle is just clearly omitted from any detections. A perception model for vehicles will easily detect this car. However, if it is re-trained with the shown bounding boxes as ground truth, it learns to disregard white cars, which would likely be a dangerous failure mode.</p><p class="ql-align-center"><img src="https://lh5.googleusercontent.com/4B7S6d1FE4cK_UP14Ma9iDx2wd7Rq2M1yHUInc7UqZKW6VXOoNUFmr_FNlsZFTiwSPlBPa-k21oGgGEegKF6AM3X5X5QZcjzpT__NNMMhLXw71IWjNrhIT8kJamypgptKEC2308JFeNFuUYpRkOi9P4"></p><p>The confusion matrix is again a good starting point to find these cases and make sure they are corrected. We can find the row with “no ground truth” in the bottom and query for examples where the model is currently still predicting an object.</p><p><img src="https://lh4.googleusercontent.com/F5tkfSRWH0rvhyDN4okzsTBgsse8qDz9KwiNe6FD-5IFbDgactKLJyEPDTZpllDriNkf4ghppUwDV6rhkU826ijXHLRQqaKsxwJ7pFY4BY2Hkda5VN2mwRJC8Ko4Ow0s0uyA15jENUHUwK-xEC2kkfE"></p><p><img src="https://lh3.googleusercontent.com/vexNOiYxXuY_0_P8Ani_ao266gUZbJFRaYxSNsuisQbZbYVgbRzlP93fXzdB7J9jREzqYnGdBAEXZ9xWaoZcNdiKfxFf9AMDIXYks2TDL1e_8FgzlBh8YXTOsqeylMdR2CgrhhQB6VcTHWjlshkNmNY"></p><p>With this query, we can swiftly find all cases that are obvious misses and make sure that the data annotations are corrected for these images.</p><h3>Dealing with long tail data distributions: the TL;DR</h3><p>We talked about two common problems when training high-performance computer vision models: unequally distributed data with rare edge cases and bad image annotations.</p><p>Both research and industry agree that picking out the right data to annotate leads to much better model performance, because it corrects for the naturally unequal data distribution encountered in the real world. The model can learn how to deal with edge cases by being trained on more data that contains them. Secondly, almost all datasets contain annotation errors, while computer vision models work on the assumption that all annotations are correct. This causes the models to perform worse, in practice. This problem is most pronounced for rare edge cases, as these are harder to annotate and they have a worse impact if they are annotated incorrectly.</p><p>There are proven methods to cope with these problems. Good strategies to pick high quality data include starting from difficult images and finding more of the same or running semantic search for situations that are known to be problematic. Second, we have learned that reviewing the quality of data annotations is essential. Using error metrics that compare annotations with current model predictions (e.g. with a confusion matrix) is a good way to quickly surface the lowest quality data labels or even missing annotations.</p><p>Getting the edge cases right is considered to be the “holy grail” of building high performance computer vision models, and with the right techniques, you don’t have to lose sleep over them any longer.</p><h3>Ready to try Nucleus yourself?</h3><p><br></p><ul><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+woman+with+a+baby+stroller" rel="noopener noreferrer" target="_blank">Find edge cases from scratch with Natural Language Search</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=image&amp;display=similarity_image" rel="noopener noreferrer" target="_blank">Find similar images from a single sample</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=bus&amp;order=Desc&amp;metric=nucleus_confidence&amp;annotation_labels=truck" rel="noopener noreferrer" target="_blank">Find mistakes in computer vision labels (i.e. class confusions)</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;explorer_patch_display=false_positive&amp;iou_extent=0%2C0&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=car&amp;order=Desc&amp;metric=nucleus_confidence" rel="noopener noreferrer" target="_blank">Find missing image annotations</a></li></ul><p><br></p>

 A guide for ML engineers losing sleep over edge cases

Don't lose sleep over long tail data distributions.

How to Handle Long Tail Data Distributions in Machine Learning

<p>Within the field of computer vision, semantic segmentation is a close approximation to describing the visual world the way humans perceive it: entities with precise boundaries. For use cases where nuanced scene understanding is mission-critical, like autonomous vehicles and intelligent robots, building models requires massive amounts of highly accurate segmentation masks with pixel-level precision.</p><p>The exactness required near object edges that makes semantic segmentation valuable also makes it incredibly time-consuming for humans to annotate.&nbsp;Given taskers annotate thousands of images per day, we wanted to accelerate the <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotation</a> process without losing fine-grain precision. To accomplish this, we developed a feature called <em>Autosegment</em>. An annotator draws a box around an object and a segmentation mask is automatically generated. The annotator can then adjust the mask before marking it as complete. Under the hood, the subset of the image is sent through a model, which automatically identifies and segments the object of interest.</p><p><a href="https://scaleai.wistia.com/medias/lvnzhzfl7b" rel="noopener noreferrer" target="_blank">video</a></p><p>The <em>Autosegment</em> functionality was originally developed for autonomous vehicle data, which is a core customer use case at Scale. (You can read more about our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">original <em>Autosegment</em> tool</a> in our previous blog post.) When it was first launched, we saw a 30% decrease in the time it took to produce segmentation masks. However, the use cases for <em>Autosegment</em> have expanded greatly since its introduction, with AV data now constituting only one of many domains for which Scale produces segmentation data. While the underlying model did generalize beyond the autonomous vehicle domain, we observed that it performed noticeably better on objects like street signs, vehicles, and pedestrians than on other types of classes.</p><p>This article contains a technical deep dive into the machine learning model that powers <em>Autosegment</em>. We’ll walk through some of the choices we made during the development process as well as our recent effort to overhaul the model for improved generalizability. In particular, we will discuss the data-centric approach we took to boost the usefulness of <em>Autosegment</em> across a wider range of use cases.</p><h2>Model Anatomy</h2><p>The segmentation model developed to power this tool addresses a problem known as <em>salient object detection</em> wherein the goal is to identify and segment the most “salient” or prominent object in an image. There is subjectivity in this task because an image can easily contain more than one prominent item. The intention behind <em>Autosegment</em> is that the user draws an approximate bounding box around the object of interest. Based on this we can firmly make two assumptions:</p><ol><li>The target object will be relatively centered within the box</li><li>The target object will extend nearly to the edges of the box, with some nonzero padding</li></ol><p>We encode these assumptions into the training process. Specifically, images in the training set are cropped in such a way that the target object is always centered with some randomly added padding. For any training image containing multiple segmented objects, each object is cropped separately to produce an individual training observation.</p><p><br></p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fe23c12266e384e7ca60a35f93258ba3e"></p><p>The architecture we chose to implement for this purpose is based on U2-Net proposed by <a href="https://arxiv.org/pdf/2005.09007.pdf" rel="noopener noreferrer" target="_blank">Qin et al.</a> for this type of problem. U2-Net provides two concrete benefits over other architectures we considered.</p><p>First, it is designed to more effectively take into account both local and global contrast features of the image. Most of the popular feature extractors (VGG, ResNet, etc.) were designed and optimized for image classification. While they can work well for segmentation tasks, they do so by identifying features specific to the semantic classes they are trying to predict. The consequence of this is that they tend not to generalize as well to classes absent in the training set. On the other hand, U2-Net is designed to make use of image features that indicate object saliency. Because of this, while U2-Net may not perform as well at outright classification as more traditional architectures (we did not test this), in our testing it does a good job at identifying salient objects pertaining to obscure classes. This is important for Scale’s customers who often need to annotate data for niche use cases not represented in the training data.</p><p>Second, we want to carefully avoid the need to stretch or downsample images. Because the goal is pixel-perfect accuracy, the model needs to output a high-resolution segmentation mask. Downsampling, stretching, or otherwise modifying the image in any information-reducing manner before feeding it to the model will typically result in blurrier segmentation masks than desired. To this end, utilizing a fully convolutional network allows for the processing of any image crop size or shape as long as it is large enough to be processed by the encoder phase of the U-Net (and small enough to fit within GPU memory). In addition, unlike some other architectures, U2-Net does not excessively downsample the input with convolution strides or pooling. This helps propagate enough information about the original image through the network to produce high-resolution mask outputs.</p><h2>Improving Generalizability</h2><p>Neural networks have a tendency to try and “cheat” by taking shortcuts when learning to solve a problem. Faced with the task of salient object detection, one shortcut is to simply learn how to segment the specific classes present in the training data. For example, if trained exclusively on autonomous vehicle camera data, the <em>Autosegment</em> model only needs to memorize what certain object classes look like, such as cars, trucks, and pedestrians. The resulting model may work for annotating vehicle scenes, but it will not perform well when applied to new object classes because it hasn’t truly learned the concept of “saliency.”</p><p><a href="https://scaleai.wistia.com/medias/dhzm8kaohk" rel="noopener noreferrer" target="_blank">Video</a></p><p>To overcome this challenge, the model needs to be given training examples for which this shortcut does not work. In the case of <em>Autosegment</em>, we expanded the training dataset to include hundreds of new object types. Neural networks have a finite number of parameters,&nbsp;meaning that there is a limit to how much “knowledge” they can hold. By providing more classes than the network can effectively memorize, the hope is that it will be forced to learn a simpler strategy that can detect saliency across a more general set of images.</p><p>Our first step towards generalization was to incorporate the open-source <a href="https://cocodataset.org/#home" rel="noopener noreferrer" target="_blank">Common Objects In Context (COCO)</a> segmentation dataset into the training process. However, while COCO contains a wide variety of object classes, it only contains about 200 thousand annotated images, which is a small fraction of the training data from autonomous vehicle cameras. In addition, COCO segmentation masks are often somewhat imprecise.</p><p>We then sought to leverage the vast segmentation data that has been produced using Rapid, our annotation platform for rapid experimentation. Rapid supports both full-scene segmentation and partial-scene segmentation. In full-scene semantic segmentation, every pixel in the image is classified into one of multiple classes. In partial-scene segmentation, pixels may be categorized into zero or more classes. Hundreds of thousands of images have been annotated using Rapid.</p><p><a href="https://scaleai.wistia.com/medias/15nwn1o6ff" rel="noopener noreferrer" target="_blank">Video</a></p><p>Full-scene segmentation often contains classes that are not beneficial to train on. For example, in outdoor scenes the sky is often a distinct class in the segmentation mask but is typically considered less salient than all other objects in the image. While we want to train on a diverse set of object classes, we do not want to train on “background classes” or objects that do not match a reasonable definition of saliency. Neural networks are generally robust to occasional adverse training samples, but in large quantities they can negatively affect training. In order to filter out these undesirable classes, we have taken the time to construct a mapping between a common taxonomy and the customer-specific classes. For example, this mapping helps us recognize that what one customer calls a “bicycle” is the same as what another customer calls a “bike.” From this, we filter out segmentations that do not match a list of approved classes. Partial-scene segmentation is the more common project type on Rapid and mostly avoids this problem of extraneous classes.</p><h2>Validation</h2><p>An important consideration for us was that because <em>Autosegment</em> is already in use across a large number of projects and use cases, changing the underlying model risks unknowingly worsening its accuracy on specific categories of images. In theory this type of accuracy loss could occur due to the finite information capacity of the model architecture. As mentioned earlier, the network needs to encode its problem-solving process using a limited set of parameters. If the number of parameters is too small, underfitting will occur and impede accuracy. We combat this possibility in two ways.</p><p>First, we ensure that the training data is well-balanced. If the overall composition of the dataset is skewed in favor of any particular domain, then the model may tend to overemphasize training samples from that domain. Not only can this result in lower accuracy in other domains, but it could impede our goal of forcing the model to learn a generalized solution to the salient object detection problem.</p><p>Second, in addition to benchmarking accuracy on the overall collection of images, we also benchmark the model’s performance on subdomains (such as autonomous vehicle data) to ensure it performs well within each of the strata. This essentially serves as “regression testing” for the model. Regression testing is a software development practice to ensure that code changes do not break existing known use cases. To accomplish this, we validate the model on independent test sets representing distinct customer use cases. The crucial use case we monitored during these improvements was autonomous vehicle data:</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fdb0470f15fa945cda7b946e690837b34"></p><p>Beyond standard pixel classification metrics, we also calculate the precision and recall of pixels within a small neighborhood of the segmentation edges. These metrics help us assess how well this tooling will actually benefit annotators for whom the most time-consuming aspect of semantic segmentation is the object boundary pixels.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fac64ee3d7a814bc0852260ed4e5721f9"></p><p>To calculate these edge metrics, we first start with the predicted and ground truth masks. Keeping only the single-pixel boundary of each mask, a <a href="https://en.wikipedia.org/wiki/Dilation_(morphology)" rel="noopener noreferrer" target="_blank">dilation operation</a> is applied with some small threshold. Metrics are then calculated from the dilated edges. The above image visualizes this process.</p><h2>The feedback loop</h2><p>Beyond the initial training of the model, data annotated with Scale Rapid continues to get fed back into the training process. This creates a feedback loop: <em>Autosegment</em> helps users to more quickly produce higher quality annotations, and in turn, those annotations serve to continuously improve the automatic segmentation accuracy. Having high-quality training annotations is an indispensable prerequisite to training a high-performing model. This <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a> lifecycle is a unique characteristic of Scale’s annotation platform and fundamental to how we achieve best-in-class annotation speed and accuracy.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F03a3cce06e0e4cfba0dd090356f2669d"></p><h2>How you can use <em>Autosegment</em></h2><p><em>Autosegment</em> is one of the multiple Active Tooling capabilities that exist in Scale’s products. Active Tools provide interactive ML-powered assistance to data annotators. By letting humans make changes to the model inputs in real time, model outputs can be quickly dialed in to match the situation precisely and minimize time spent on the task.</p><p>You can supercharge your data annotation using <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a>, which offers best-in-class annotation infrastructure to accelerate your in-house annotation team. Studio also offers a free tier to familiarize yourself with the platform and start annotating data. If you’ve decided you’d like to offload more parts of the data annotation pipeline, <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> provides an easy way to send your data to be annotated by Scale and get production-quality labels with no minimums.</p>

How Scale AI’s computer vision annotation tooling (CVAT) uses machine learning-assisted tooling for faster and more accurate semantic segmentation.

Machine Learning-Assisted Image Semantic Segmentation

Machine Learning Assisted Image Segmentation

Learn how we extended our infrastructure by sharding our MongoDB cluster.

From MVP to Serving Enterprise Scale

How We Scaled Our MongoDB Instance Gracefully

<p>Large-scale datasets are at the center of model development and evaluation. Quality of training data is the dominant factor for model performance. Anyone can create an excellent model based on great data, but even the best ML engineers can’t design good models with bad data.&nbsp;</p><p>If you’ve ever used popular public datasets such as ImageNet or MNIST, you would assume that the class and attribute <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">labels</a> are correct. Shockingly, ImageNet, the most widely used dataset for popular image recognition systems, has a 6% error rate[6]. When models are fed bad data, such as incorrect classes, incorrect features, or missing <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a>, they often have severe classification problems, which can destabilize ML benchmarks and contribute to representational harms in datasets.&nbsp;</p><p>Examples of representational harms[1] in datasets include under-representation of darker skinned subjects compared with lighter-skinned subjects within prominent facial analysis datasets[2,3], and millions of images of people in ImageNet labeled with offensive categories, including racial slurs and derogatory phrases[5].&nbsp;</p><p>Even in leading applied ML research groups, approaches for characterizing and fixing labeling errors in massive datasets are limited.&nbsp;</p><p>One approach to combat noise in a dataset is Confident Learning (CL), which is based on pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. However, CL entirely ignores the case where one or more true labels are missing. This limitation occurs because CL relies on building a pairwise class-conditional distribution between the given and true labels[6]. Other techniques include sorting by model loss, which surfaces the examples that are most ambiguous and difficult to learn, and sorting by confidence where model and ground truth disagree (quite effective!).</p><p>What is the most principled option after amassing a collection of bad labels? In practice, it would be fixing your dataset by going through the images and fixing the <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a> one by one.&nbsp;</p><p>This is both laborious and impractical.</p><p>That’s why (1.), a way to visualize, triage, and organize your data, and (2.) a way to fix your annotations, are critical to your data annotation project.</p><p>Scale provides an integrated dataset and workforce management platform that solves these problems.</p><p>Let’s walk through an example of how you can use two of Scale’s products: Scale Nucleus and Scale Studio to identify and fix bad labels leveraging model predictions. For this example, we’ll be using the public <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a>.&nbsp;</p><p><img src="https://lh3.googleusercontent.com/hYW3_C_Yz4vuaZKQppVBRJB273v9Nz2b9bgHBDbl-WUrdKjfHg31Ic6E2j34WJZSuXSufVkBgfWf0Q7ThfnZ-l7CxScpOViGECQXXHBhk3_VkMqYT0t-GpgGL_1J_O97Vg7wmCjTPh-PSodX-Zm7o-c"></p><p>Starting on the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=insights&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg" rel="noopener noreferrer" target="_blank">“Charts” page in Scale Nucleus</a>, a simple click into the confusion matrix shows 2,277 cases in which models predicted “person” but the ground truth label is “car”.</p><p><img src="https://lh5.googleusercontent.com/e-ElAIxlS6X1koxM2vOVAHv-MUG_IC_trcCX6K1YXyvDDPZbO4RnNOXBXRUgUaK6Apf6c9AM5-NHq67tounEbnyEqoQbMxKoxnRndm3FFaMh4CNkTbrbR7qNg_W1p5JWwvD9DuNViLj2BUt1NfahoZg"></p><p>After sorting the results by descending model confidence, a brief scan of the corresponding objects makes it clear that these annotations are wrong.</p><p>This may be surprising given that the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a> is being used by thousands of researchers and models, but almost all datasets have these kinds of issues.</p><p><img src="https://lh4.googleusercontent.com/iPxyYxuBrw-c2817ZQiLrCxCU_xVEyC4aHDq5044UO30_qAEeXOXc3H9PnsXxiaD9BLRI4reFCGERZ1eI4KTjVRmoCifT41hOh8mGIvOOBMwwMJhG8oQuEE2aE5iiExfkkoRRu2LLbFxiDO3OkoWYeo"></p><p>Now that you have discovered some errors with the data labels, you might want to correct them.</p><p>With one click, you can send the bad labels over to Scale Studio, our self-service annotation platform, where your workforce can review and fix the errors. The annotation results will be automatically imported back into Scale Nucleus, overwriting the previous, flawed set of labels.</p><p><img src="https://lh3.googleusercontent.com/JWVK3_yCVI-Z1udBltlndb0T3qWfKboWTrHFaW2IFYnIoggE2PojskkIvJfQgBecKtkJtzf5f2ti7QeYrS7Cog2mPdnhHItx2-e31Uh3ZV3HM3fqcBTBQSpSrUTcCH9cTqvZphpSoO2ulpwyOLhB-o4"></p><h2>What’s next:</h2><p>At Scale, we are perpetually focused on both data and label quality. That’s why we build tools, products, and platforms to ensure that you’re getting the best results you possibly can from your ML models. But building high-performance models starts with error-free datasets. As you are <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">exploring your dataset in Nucleus</a>, verifying label accuracy, identifying edge cases, and collecting mismatches between predictions and ground truth, you may find that you and your teammates want to collaborate on adjusting or improving your ground truth labels. If you want to make these edits in house, in the most efficient way possible, <a href="https://dashboard.scale.com/signup?product=studio" rel="noopener noreferrer" target="_blank">sign up for Scale Studio today</a>.</p><h2>References</h2><ol><li>Crawford, Kate. "The trouble with bias." <em>Conference on Neural Information Processing Systems, invited speaker</em>. 2017.</li><li>Buolamwini, Joy, and Timnit Gebru. "Gender shades: Intersectional accuracy disparities in commercial gender classification." <em>Conference on fairness, accountability and transparency</em>. PMLR, 2018.</li><li>Raji, Inioluwa Deborah, and Genevieve Fried. "About face: A survey of facial recognition evaluation." <em>arXiv preprint arXiv:2102.00813</em> (2021).</li><li>De Vries, Terrance, et al. "Does object recognition work for everyone?." <em>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops</em>. 2019.</li><li>Crawford, Kate, Trevor Paglen, and A. I. Excavating. "The politics of images in machine learning training sets." <em>Excavating AI</em>.</li><li>Northcutt, Curtis, Lu Jiang, and Isaac Chuang. "Confident learning: Estimating uncertainty in dataset labels." Journal of Artificial Intelligence Research 70 (2021): 1373-1411.</li></ol><p><br></p>

Nucleus helps you curate data to train better models; Studio helps you use internal resources to improve your dataset in ways that Nucleus flags.

Sometimes it's helpful to bring your own labelers domain expertise to assist in both data and model quality.

Building Great Models with Better Data on Nucleus and Studio

<p>Nucleus is the centralized, scalable, and collaborative control center for your machine learning (ML) model’s training data. Nucleus helps you update missing or erroneous labels, and identify model failure cases so that you can direct your data curation efforts with the express goal of training better models.</p><p>With simple integrations for model integration, testing, validation, and deployment, Nucleus can serve as your one stop shop to continuously iterate on your dataset for superior model quality, even in the face of new edge cases, model drift, and unexpected changes in environmental variables. Nucleus helps you tackle the long tail, the one certain path to training models that can address new scenarios and eliminate failure modes.</p><p>Read on to learn the essentials—how Nucleus can help you manage your dataset efficiently.</p><h2>The basics:</h2><h3>Addressing model failures:</h3><ul><li><em>Where is my model currently failing and how can I fix these issues with the next round of labeling data?</em></li></ul><p>Once you’ve uploaded ground truth labels and at least one set of predictions for your dataset, you can click into the “Objects” view in Nucleus to identify False Positives, as shown in the below video. You can also identify other simple failure modes like False Negatives or low Intersection-over-Union (IoU) scores for image data.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/511bisntyb" rel="noopener noreferrer" target="_blank">this video</a></p><h3>Addressing label quality issues:</h3><ul><li><em>How can I easily check and improve the quality of existing labels?</em></li></ul><p>With your ground truth labels already in Nucleus along with one set of model predictions, the IoU slider and the “sort by” drop down menu lets you sort classifications by confidence from low to high. You can then choose if you want to label or fix the lowest-confidence images, with Rapid or if you’re an enterprise customer, your labeling pipeline.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/bgmcf03b9u" rel="noopener noreferrer" target="_blank">this video</a></p><h2>Breaking down your data in new ways:</h2><h3>Finding images without labels:</h3><ul><li><em>How can I find images with certain properties for which no ground truth label is available?</em></li></ul><p>If no label exists for the query you’ve conceptualized, you can search your dataset with simple, natural language text. Using CLIP embeddings under the hood, Natural Language Search lets you search for images with English phrases, instead of the structured query language you use for labels and metadata.&nbsp;</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">this video</a></p><h3>Finding similar images:</h3><ul><li><em>I’ve identified an example image: how can I find all similar images in my dataset?</em></li></ul><p>If no label is present for a class of images, but you’ve found a sample image, you can use Object or Image Autotag to build up a Slice of your dataset that consists of similar images. A step-by-step process will show you how to provide positive and negative examples, in multiple rounds, to fine-tune your search for like images. When you proceed to query your Autotag, you can set a threshold to broaden or narrow the scope of your search.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/uuavyfeapq" rel="noopener noreferrer" target="_blank">this video</a></p><h2>Debug your models:</h2><h3>Find classification failures:</h3><ul><li><em>How can I find examples of classes in which my model and my labels disagree?</em></li></ul><p>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can find a Confusion Matrix on the Charts page. This grid lets you find color-highlighted cases in which the labels and model disagree on a classification. You can click on any box to filter your entire dataset on just those examples, to further explore where your model fails and where it succeeds.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/ic8y27jymv" rel="noopener noreferrer" target="_blank">this video</a></p><h3>Sorting and filtering both failures and successes:</h3><ul><li><em>How can I find the best and worst classification examples?</em></li></ul><p>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can sort all of your data objects by confidence or intersection-over-union (IOU), or bound your images with specific minimum and maximum thresholds for these score values. You can also filter by specific types of confusions such as False Positives and False Negatives.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/c2f14217wn" rel="noopener noreferrer" target="_blank">this video</a></p><h3>Avoiding regressions:</h3><ul><li><em>How can I ensure I'm not breaking anything that worked with a previous model or dataset?</em></li></ul><p>As you improve and expand your dataset, then proceed to train new models, you’ll want to ensure that your new models don’t fail on cases that your old models solved successfully. Through Scale Validate, Nucleus helps you track metrics such as mean average precision (mAP), precision, and recall on specific scenarios in your dataset. Or, you can establish and write your own metrics to track over time. This process ensures that every label addition and dataset modification you make only yields improving outcomes rather than regressions. You can think of this as a continuous integration (CI) or regression detection process for your models.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/ak5lqayith" rel="noopener noreferrer" target="_blank">this video</a></p><p><br></p><p>Ready to try Nucleus? <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Click Here</a> to set up your first project.</p>

Nucleus facilitates data curation and dataset management.

Learn how Nucleus facilitates dataset curation, to help you and your team train and debug better models.

Nucleus Simplified—Dataset Management for Everyone

<p>All companies, from the earliest startups to the largest Fortune 100 corporations, face a decision as they scale up their artificial intelligence (AI) and machine learning (ML) efforts: How much of it should be built in-house?&nbsp;</p><p>That’s a particularly thorny question when it comes to <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a>. It’s important to get the approach right from the start; studies have shown that an abundance of high-quality annotated data is typically the most significant driver of model performance. But <em>how</em> it’s done, across all steps of the data process — from selecting a <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a> platform, to writing high-quality instructions and taxonomies, and doing the annotation itself — must be based on what works best for each unique organization.&nbsp;</p><p>Here we explore what to keep in mind — the benefits and hidden pitfalls at each step of the process — as you try to solve your own data annotation build vs. buy equation.</p><h2>Selecting a Data Annotation Platform</h2><p>Choosing the right data annotation platform from the beginning of the process can minimize complications arising from changing approaches or platforms in the middle of a project.</p><p>As you think about a software “home” for your data annotation activities, there are a few key capabilities you should consider:&nbsp;</p><ul><li>How easy is it to set up a new project or pipeline for data annotation?</li><li>How quickly and easily can annotators add high-quality, accurate annotations to your data?&nbsp;</li><li>How simple is the process to add annotators to the platform?&nbsp;</li><li>Can you analyze the annotation workforce’s performance across dimensions such as throughput, quality, and time spent?&nbsp;</li><li>What are the guardrails or in-product capabilities to ensure the data annotation output is as high-quality as possible?&nbsp;</li><li>Can you import your model output as “hypothesis annotations” to give your annotators a head-start on their work?</li></ul><p>Indexing on these capabilities can help guide your decision on whether to build a platform in-house or purchase a software solution. Once you’ve established your criteria in each of these dimensions, the build-vs.-buy equation comes down to a few ROI calculations:</p><ol><li>Customizability: If you’re looking for a platform tailor-made to suit your specific annotation needs, there will be no platform that is more customized than one that you and your team build yourself. This is doubly true if you have annotation needs that are non-standard and have minimal (if any) support from annotation platforms on the market today. The fact of the matter is, commercial tooling caters to the most common use cases, workflows, and annotation types, which may suit the majority of situations, but may not work for you. Some people customize an open source solution (such as <a href="https://github.com/openvinotoolkit/cvat" rel="noopener noreferrer" target="_blank">CVAT</a>) in order to suit their needs, and others build their own from scratch.&nbsp;</li><li>Predictability: Are your AI/ML efforts predictable over the next few years, or is it likely that you’ll experiment with different data types or annotation volumes? If you expect the scope of any AI/ML project to evolve rapidly, buying a commercial platform can be a prescient choice, since you’ll be able to spin up new data types or annotation types more quickly, provided that they’re supported. If you have a sense of data types you’ll likely experiment with in the future, it’s smart to check if they are supported in whatever annotation platform you consider</li><li>Time to market: If speed to production is an important consideration, buying an annotation platform will almost always get you there faster than building one in-house. The one exception is if your data and annotation type are simple, and supported by an open-source solution out-of-the-box. Otherwise, building a custom in-house platform can take months to get the first project in production. That’s an opportunity cost that is often neglected, but which may pose one of the greatest implicit expenses (or loss of revenue) for your organization.</li><li>Resources for maintenance: Building an in-house solution will require engineers to maintain and update it to accommodate new experiments or pipelines, and keep up with new standards and approaches. That said, even if you choose a commercial platform, you’ll still likely need to budget some engineering time to set up your pipelines – albeit less than if you’re starting from scratch – especially if you are utilizing API capabilities to import data, create projects, and/or export data.&nbsp;</li></ol><p>In your overall consideration, keep an eye on both short-term and long-term costs. For some smaller organizations, building an in-house tool or customizing an open-source solution “feels” cheaper because there’s no explicit cash outlay. However, you should consider whether the time and expense involved in establishing and maintaining your own data annotation tool is worth it. Future budget demands are more likely to go toward developing and training models than toward maintaining your in-house data annotation platform.</p><p>For the largest organizations with custom annotation needs, the calculus becomes more nuanced. In these cases, where the team has a need for steady, large-scale annotation, where having a team of engineers dedicated to building and maintaining an in-house platform could start to make financial sense. That said, if you anticipate annotation needs evolving and quick time-to-market is important, be sure to still consider a commercial solution – at least initially, with an eye towards a potential eventual transition to in-house.</p><h2>Developing High-Quality Instructions &amp; Taxonomies</h2><p>In addition to selecting the right platform, you will also need high-quality instructions and taxonomies if you want to train high-quality models. Whether you use your in-house annotation team or outsource to a third party, explaining exactly what you want annotated (along with examples of what to do vs. not do) is critical to getting high-quality data back.&nbsp;</p><p>This is an area where it makes sense to keep in-house as much as possible. You and your team will have the best, most intimate knowledge of what exactly you need annotated, as the team developing the models. By keeping this in-house, you’ll have a tight feedback loop as you scale up. You’d be surprised at how certain instructions are interpreted — and by having your ML team stay tightly involved in workshopping the instructions, providing examples, and adjusting the taxonomies, you can be confident that you're getting the best possible data back.</p><h2>Choosing a Workforce for Data Annotation</h2><p>Most teams start annotating data with internal employees who already are familiar with the data and who can start the process part-time. But as the data volume ramps up, be prepared to either recruit new full-time members to the annotation team or to outsource the work entirely to a third party.</p><p>Things to consider when making this decision:</p><ul><li>Cost: Paying the annotation team is not the only consideration here. For a more realistic assessment of the cost of outsourcing data annotation, factor in the time and effort involved in managing an in-house team.</li><li>Flexibility: Will your data annotation needs ramp up and down quickly? Some third-party providers are prepared to handle such surges with minimal disruptions. An in-house team will need to be kept prepared to jump back in during a scaleback, while still appropriately occupied.</li><li>Subject matter expertise: While some third-party providers offer teams with specific expertise, most do not. If your annotation work is highly specialized, it may be best to recruit and train your own data annotation experts in-house.</li></ul><p>Some organizations choose to utilize a blend of in-house as well as external data annotators. One common workflow is using the external team to do the ‘first attempt’ of annotation, while the internal team reviews the attempts and provides an additional layer of quality assurance. Others utilize the external team to annotate steady production pipelines, and the internal team to work on data for short, bursty experiments.</p><h2>Conclusion: Buy Some, Customize Others</h2><p>Ultimately, answering the build vs. buy question doesn’t have to be an all-or-nothing proposition. If your data annotation needs are particularly custom, your budget for in-house development relatively unrestricted, and any future needs relatively predictable, you may be able to develop a great annotation solution in-house. For other teams, commercial software is likely a good choice: it can be deployed rapidly, and you can choose how and when to use its various parts. Depending on your needs and scope, third-party providers can work with your in-house teams to build or modify features and extensions as your AI/ML projects evolve.&nbsp;</p><p>If you’ve considered your options and decided that you’d like to try an annotation platform built by annotation experts, <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> offers best-in-class annotation infrastructure to accelerate your in-house annotation team. Studio also offers a free tier to familiarize yourself with the platform and start annotating data. If you’ve decided you’d like to offload more parts of the data annotation pipeline, <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> provides an easy way to send your data to be annotated by Scale and get production-quality labels with no minimums.</p>

Choosing the right data annotation approach can minimize complications down the line. What are the factors you should consider in making this build vs buy decision?

Should you build or buy a data annotation solution?

Build vs. Buy: What Makes Sense for Data Annotation?

<h3>It all started with a breakdancing video.</h3><p>Now, on the face of it, that shouldn’t have been too surprising, back in 2010. YouTube was already well on its way to assembling a massive collection of the human experience. We had run programs like Life in a Day - asking viewers around the world to submit videos of their daily life on a single day - to document everyday life in a way no-one had parsed before. Breakdancing and hip-hop culture were undeniably a part of that global experience - maybe surprisingly, in countries like France, where communities like <a href="https://fr.wikipedia.org/wiki/Juste_debout" rel="noopener noreferrer" target="_blank">Juste Debout </a>had started to document a thriving scene.&nbsp;</p><p>But at YouTube, we had a big problem making sense of those videos. Viewers from around the world had uploaded a huge range of content, with little in the way of titles or tags to make sense of it all. For many uploaders, it was their first time uploading a video to the internet - <em>ever</em>. In spite of that, what made YouTube successful, and what my team focused on, was a vision to drive viewers to videos they didn’t expect, and that hadn’t amassed thousands of views - through related videos, through search, and through trends.</p><p>So you can imagine my surprise in 2010 when my team showed me a 15 second video of a man spinning on his back with his sneakers in the air, on a street somewhere in Europe. The video had a random title and the uploader had no followers. Under normal circumstances, this would be a very difficult video for someone to find - just a random selection out of hundreds of thousands of videos that morning. But that day, we tried a new approach to understanding videos, based on what were called Convolutional Neural Networks, with the capacity and training horsepower to work across the whole range of the YouTube catalog.</p><p>Sure enough, this time, bubbles appeared next to the video, saying “breakdancing”, “spinning”, even “<a href="https://www.google.com/search?q=breakdancing+munchmill&amp;rlz=1C5GCEM_enUS996US996&amp;ei=hCuWYtHoM7m2uvQPy9eCcA&amp;ved=0ahUKEwjRuq_M_on4AhU5m44IHcurAA4Q4dUDCA4&amp;uact=5&amp;oq=breakdancing+munchmill&amp;gs_lcp=Cgdnd3Mtd2l6EAM6BwgAEEcQsANKBAhBGABKBAhGGABQ9QJYpA1gkRFoAXABeACAAWyIAfYGkgEDNC41mAEAoAEByAEIwAEB&amp;sclient=gws-wiz" rel="noopener noreferrer" target="_blank">munchmill</a>” . Trained on tens of millions of videos, deep learning had discovered the connections between this human movement across a few frames of video, and an entire global culture built around hip-hop.&nbsp;</p><p>For me, those connections launched the next decade of my career. The combination of powerful, expressive machine learning models, with new approaches to understanding the diversity and complexity of content shared on the web, was an incredible opportunity to build new experiences. My time launching a startup with Steve Chen highlighted for me how those techniques could be brought to bear to live and immersive content, while my time at Apple helped me to understand how “data centric” AI could also be used to build new hardware and software experiences with a “privacy-first” mindset.</p><p>I recently joined Scale to run our engineering organization, in part because of a similar credo that Scale has in our culture - that Ambition Shapes Reality. The next decade of AI achievement is going to be built by teams that take on outsized ambitions, and understand that it is not always a straight line to get there.&nbsp;</p><h3 class="ql-align-center">Operational&nbsp; &amp; Data Centric AI</h3><p>&nbsp;The market for AI developer tools and infrastructure is currently small, and limited by the number of companies who are actually AI capable and have teams capable of using ML to drive business goals. But in the long run this market will be an industry-defining set of capabilities, and we have a golden opportunity to own the primitives that build this market, given our leadership position in autonomous vehicles and perception data. Those capabilities will include the ability to drive operational improvements through a counterfactual view of complex systems, being able to leverage multi-modal and Foundation models to work across visual and language data, and building insights in real-time as data distributions shift.</p><p>Building out these capabilities for ML teams not only gives us a call option on this really exciting future, but it also gets us early and close access to the small pool of AI practitioners who are pioneering that future today. That partnership between forward-looking researchers and product leaders and engineers characterizes the fastest-moving ML teams today.</p><h3 class="ql-align-center">Running Through Walls</h3><p>When I was thinking about joining Scale, I realized I had a lot of vision alignment with Alex, Scale’s CEO. In an internal memo, Alex recently wrote, “I am tremendously excited about what’s beginning to really work here, probably more than I’ve ever been in the history of Scale.” It was obvious to me that Scale is unique, in part due to the massive opportunities in the rapidly growing emergent field of AI, and also in part by the mission upheld by leadership.&nbsp;</p><p>Alex truly understands the nuances of AI. He recently did a <a href="https://www.youtube.com/watch?v=diBoo0pQ8c8" rel="noopener noreferrer" target="_blank">TED Talk at Berkeley </a>where he reiterated that AI needs humans to teach it individual values, nudge it to find thoughtful outcomes, and ensure that human intentions and values are aligned with the AI. He talked about how machines make mistakes and how they have to be taught to tell the truth.&nbsp;</p><p>This alignment was a grounding factor for me. Alex understands that “AI is a supercharger for humans - when AI is better than humans, it makes humans better. AI will enable us to be even more creative and more idea driven. It allows us to embrace the generative aspect of human nature so we can run faster with our ideas and build better, more powerful solutions to the world’s biggest problems”. And building solutions to the world’s biggest problems is a focal point of Scale’s culture and formed the company’s Credos.</p><p>Our credos provide a framework for us to make decisions and work effectively as a team. The one that has resonated with me the most is “run through walls.’ We are undaunted by barriers and race towards overcoming them. We step outside of our comfort zone, test our ideas, and do the hard work to get to the right solution to a hard problem. I have seen this credo in action with every one of my team members.&nbsp;</p><h3 class="ql-align-center">Become a Scalien!</h3><p>Scale is in an exciting place where growth opportunities for AI organically grow your career too. Being at Scale allows you to work with high visibility customers - often impacting millions of users.</p><p>You’d have the opportunity to ideate with some of the world’s leading ML and Data Science teams at our partner companies, and find creative ways to adapt new ML techniques like Generative Models, Diffusion Models,&nbsp; and Large Language Models, or Online Optimization techniques applied to expressive embedding spaces, to building a new product.</p><p>It’s been a long and rewarding journey since that first breakdancing video. I know AI and its innovation will continue to surprise me and I am excited to experience that here at Scale. I hope that you will join me here too.&nbsp;</p>

Blog