Scale AI is proud to announce a partnership with the Center for Strategic and International Studies (CSIS), a nonpartisan think tank focused on international security issues. Together, the organizations will explore the use of large language models (LLMs) to provide new insights and solutions for complex global challenges and foster cross-agency coordination. Together, Scale and the <a href="https://www.csis.org/programs/international-security-program/futures-lab" target="_blank" rel="noopener noreferrer">Futures Lab</a> at CSIS will build novel data sets, fine-tune curated data and advance our understanding of national security topics, such as global strategic competition that Scale and CSIS deeply value. The first project will explore LLM applications across a number of problem areas including strategic-level wargaming, domain-specific decision support and adjudication, and the development of an international relations - strategy &ldquo;Turing Test.&rdquo;&nbsp;
Over the past few years, AI capabilities, including LLMs, have advanced tremendously, enabling novel applications across many fields. However, the potential to apply these innovations to international affairs remains largely untapped. Through this strategic thought partnership, Scale will lend its expertise in AI to support CSIS's mission-driven research and analysis by providing access to Donovan, Scale&rsquo;s AI-powered decision-making platform. The platform will help its users understand, plan, and act in minutes, instead of weeks. Scale will also provide access to its Data Engine to support LLM fine-tuning to advance national security decision-making.&nbsp;
Scale will drive the collaborative development of customized LLMs to analyze massive datasets related to national security challenges, which are geared towards global strategic competition, include threats from cyberattacks, coordinated disinformation campaigns and black-box AI developments from foreign adversaries. Scale&rsquo;s forthcoming models will help uncover subtle patterns and connections that traditional, older methods may overlook.&nbsp;
&ldquo;The time is now for Scale and CSIS to join forces, given the gravity of national security concerns today,&rdquo; said John Brennan, General Manager of Scale, Public Sector. &ldquo;We are proud to combine our technology and experience with the independence and wisdom of a longstanding national security leader like CSIS. Together we share the goal of advancing democratic values, which will guide our careful exploration and experimentation with large language models for national security.&rdquo;
Researchers from CSIS plan to apply subject matter expertise to align generative AI responses with national strategic objectives and international norms. By leveraging CSIS&rsquo;s expertise, Scale and CSIS will develop generative AI that fosters a collective understanding of how AI can, and should, support national security challenges.&nbsp;
"Artificial intelligence has game-changing potential if applied thoughtfully to foreign policy and national security concerns," said Dr. Ben Jensen, Senior Fellow at CSIS. "We are eager to bring Scale AI&rsquo;s leading AI capabilities together with our team's expertise in international affairs. This collaborative partnership will allow us to better understand world issues and offer data-driven recommendations to support the U.S. national strategy in an increasingly complex world."

Scale AI & CSIS partner to explore LLMs for global challenges, strategic wargaming, decision support, & international relations. #AI #Security

Scale AI Announces Partnership with Center for Strategic and International Studies 

As the leading test and evaluation partner for frontier AI companies, Scale plays an integral role in understanding and safeguarding large language models (LLMs).
Our commitment to fostering safe, secure, and trustworthy AI is the bedrock of our mission. In August, we outlined <a style="background-color: transparent;" href="https://scale.com/guides/test-and-evaluation-vision" target="_blank" rel="noopener noreferrer">our vision for testing and evaluation of frontier models</a> and introduced a <a style="background-color: transparent;" href="https://scale.com/llm-test-evaluation" target="_blank" rel="noopener noreferrer">platform</a> designed to establish an industry-wide scalable framework, mitigate bias, and uphold the principles of Responsible AI. In September, we proudly <a style="background-color: transparent;" href="https://scale.com/blog/scale-white-house-voluntary-commitments" target="_blank" rel="noopener noreferrer">signed onto</a> the Biden-Harris Administration&rsquo;s voluntary commitments to ensure that AI is safe, secure, and trustworthy.&nbsp;
Today, we're thrilled to further our commitment to this cause, unveiling our new frontier research effort, the &ldquo;Safety, Evaluations and Alignment Lab&rdquo; (SEAL). This cutting-edge initiative is dedicated to building robust evaluation products and tackling the challenging research problems in evaluation and red teaming. Summer Yue, the former RLHF research lead for Bard at Google DeepMind, will be joining Scale to lead SEAL as Director of Safety and Standards.
Our plan to build a robust test &amp; evaluation platform
Typically, AI companies establish their safety guidelines and evaluation criteria in-house. Many frontier model developers collaborate with Scale to have experts meticulously apply these criteria while conducting evaluations. However, we've identified some limitations and inefficiencies in this approach:
<ul>
<li>Creating guidelines from scratch and ensuring consistent application of those guidelines is a labor-intensive task, which slows down the pace of progress for AI companies.</li>
<li>These guidelines, often devised without extensive community feedback, may overlook crucial safety aspects.</li>
<li>Some safety requirements tend to be duplicated across companies, with slight variations in guidelines resulting in inconsistent safety evaluation results and a lack of transparency.</li>
</ul>
Likewise, every enterprise and government deploying an LLM application is facing the need to adopt and comply with the forthcoming standards and regulations that will be put in place. Integrating each of these rules will require a detailed evaluation process that includes the creation of clear guidelines and the management of human evaluators to ensure they are straightforward to implement.
Our plan is to create a suite of broadly applicable evaluation products on our T&amp;E platform, initially addressing the most prominent and clear-cut potential safety issues outlined in <a style="background-color: transparent;" href="https://www.whitehouse.gov/briefing-room/statements-releases/2023/10/30/fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-trustworthy-artificial-intelligence/#:~:text=The%20Executive%20Order%20establishes%20new,around%20the%20world%2C%20and%20more" target="_blank" rel="noopener noreferrer">the Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence</a>, such as AI enabled dangerous biological materials, cybersecurity risks, deceptive AI generated content, etc.
In pursuit of this goal, Scale is committed to ramping up investments in advanced red teaming research and evaluation methodologies. We plan to work in collaboration with government stakeholders, standards setting organizations, our current partners in frontier model development, and alongside the broader AI safety community. We believe that sharing openly about our progress and engaging with the research community will be crucial as we refine our approach to deliver the most reliable and valuable evaluation products.
We recognize the importance of catering to the unique requirements of different products and are dedicated to continuing our customized evaluation offerings to meet the specific needs of our customers.
SEAL: Scale&rsquo;s Safety, Evaluations and Alignment Lab
We are excited to introduce the Safety, Evaluations, and Alignment Lab (SEAL), our first major research initiative aimed at enhancing transparency and standardization in the safety and compliance processes associated with LLM deployments.
SEAL&rsquo;s primary objective is to work in collaboration with standard-setting bodies to establish benchmarks for deploying Large Language Models (LLMs), by developing comprehensive evaluation products and advanced red teaming solutions. The SEAL research team is set to undertake various projects, including:
<ul>
<li>Designing and implementing robust evaluation benchmarks. Measuring and improving eval reproducibility and reliability.</li>
<li>Researching and implementing cutting-edge model assisted red teaming methods.</li>
<li>Developing state of the art LLM-based automated rating systems leveraging the Scale ecosystem for customized training data.</li>
<li>Developing rater assist techniques such as critique modeling for improving rating quality.</li>
</ul>
We are actively seeking experienced ML researchers to join us in shaping the landscape for safety and transparency for the entire AI ecosystem. If you are passionate about making a difference, apply to join the SEAL team <a style="background-color: transparent;" href="https://scale.com/careers/4336784005" target="_blank" rel="noopener noreferrer">here</a>.

SEAL: Scale’s Safety, Evaluations and Alignment Lab

Our plan to build a robust test & evaluation platform

This past month has perhaps been the most consequential for AI governance to date. President Biden unveiled his Administration&rsquo;s long awaited Executive Order (EO), the Office of Management and Budget (OMB) released their Federal Agency AI Implementation Memo, the G7 released their AI Guiding Principles, and global leaders attended the first ever Global AI Safety Summit in the UK. Scale is pleased to support all of these, which build on two previous Executive Orders and bipartisan pieces of legislation, months of Congressional hearings, Insight Forums, and other events that have started to shape the United States&rsquo; approach to AI governance.&nbsp;
&nbsp;
As Vice President Kamala Harris said during her remarks at the EO event on October 30, &ldquo;Let us be clear: When it comes to AI, America is a global leader. It is American companies that lead the world in AI innovation. It is America that can catalyze global action and build global consensus in a way that no other country can.&rdquo; This vision can only be realized if the United States&rsquo; governance framework maximizes innovation while putting in place the proper guardrails. The EO and OMB Implementation Memo are foundational steps that advance that vision in four key ways:
&nbsp;
<ul>
<li>First, these efforts recognize the critical role that thorough red-teaming and use-case specific test and evaluation will play in safely deploying AI. Scale has long supported and has worked across industry to pioneer the techniques for red teaming and test and evaluation. Recently we released our <a style="color: rgb(17, 85, 204);" href="https://static.scale.com/uploads/6019a18f03a4ae003acb1113/test-and-evaluation.pdf" target="_blank" rel="noopener noreferrer">methodology</a> that includes red teaming to expose unintended vulnerabilities, followed by a risk-based approach to test and evaluation to ensure AI that is safe to deploy.&nbsp;</li>
</ul>
&nbsp;
<ul>
<li>A risk-based approach aligns the level of risk associated with the anticipated use and an appropriately rigorous test and evaluation process. As the OMB Memo points out, certain uses like auto correction of text pose a very different risk than AI being used for medical diagnosis, and their safety assessments should be treated as such. Paired together, these two techniques provide the confidence necessary to rely on AI for specific tasks.&nbsp;</li>
</ul>
&nbsp;
<ul>
<li>The EO highlights the necessity of consensus-based standards to put in place a repeatable and robust process for test and evaluation. Specifically, the National Institute for Standards and Technology (NIST) is tasked with launching new working groups, including industry experts, to write them. Once completed, these standards will be critical to reference in future rulemakings, legislation, and ultimately underpin the deployment of safe AI systems.&nbsp;&nbsp;</li>
</ul>
&nbsp;
<ul>
<li>Second, the EO and OMB Memo recognize that data is a critical asset for the government and directs agencies to prioritize AI-ready data. This is important because AI-ready data is the key to unlocking the power of AI. The government recently released a report with more than 700 different government AI use cases, but the data infrastructure to support them must be in place for them to be successful. AI is only as good as the data that powers it, and without the proper data infrastructure in place, the government&rsquo;s ability to efficiently adopt AI will be limited. The Department of Defense has long recognized the value of AI-ready data and has called for it within their own data strategy, and the OMB memo calls it a &ldquo;National Asset.&rdquo;&nbsp;</li>
</ul>
&nbsp;
It is critical that agencies and Congress prioritize both AI-ready data strategies and devote adequate resources to carrying out those strategies.&nbsp;
&nbsp;
<ul>
<li>Third, the EO and OMB memo encourage agencies to use generative AI in ways that are safe and responsible for their day-to-day use. Generative AI applications like ChatGPT have taken the world by storm over the past year, and there is no shortage of use cases that would enhance the government&rsquo;s daily work. For example, a large language model at the Department of Energy would drastically speed up the time it takes to understand the electrical grid for next generation transportation technologies. Encouraging agencies to embrace the technology responsibly is a step in the right direction, and when paired with test and evaluation, will help the government address data overload&ndash;one of the most pressing challenges that the government faces.&nbsp;</li>
</ul>
&nbsp;
<ul>
<li>Fourth, the EO establishes &ldquo;Chief AI Officers&rdquo; at every agency. As the use case inventory referenced above notes, there are no shortage of potential AI use cases at each agency, and this new position would enable a leader responsible for ensuring that each agency truly embraces each one of them. Beyond identifying use cases, this person will also have the critical task of leading the efforts on AI-ready data, building out a team that works throughout the agency to train staff on the benefits of AI, and also understanding the funding necessary to carry out these actions.</li>
</ul>
&nbsp;
This is a critical time for the future of U.S. leadership in AI, and the release of these documents is just the starting point. Scale calls on the Administration and Congress to continue prioritizing AI over the coming months because there will be no shortage of work needed across the U.S. government. This includes implementing these new requirements, kicking off rulemakings, and developing the underlying standards to serve as the foundation for them.&nbsp;
&nbsp;
Lastly, Congress still has an important role to play. Scale looks forward to continuing to work with bipartisan leadership in the House and Senate to ensure that we craft legislation that codifies provisions from these documents, fund them where necessary, and fill the gaps that still exist.

Last month was significant for AI governance: Biden's EO, OMB's Memo, G7's Principles, and the Global AI Safety Summit. Scale supports these efforts.

Scale AI Policy Framework: Key Elements to Ensure American Leadership in AI 

RAG expands Generative AI's applications by injecting external info into prompts, aiding context-specific output generation for diverse data sources.

Enhancing LLMs with Retrieval Augmented Generation

For the past 12 years, I've been consumed by the world of data. I've taken every chance to expand my skills, from data warehousing and analytics to cutting-edge software solutions. It's been immensely rewarding to push the boundaries of what's possible with data.&nbsp;
However, despite my love for big data platforms, I no longer felt the same level of intensity and inspiration I once had. What once was unchartered territory became business as usual.
Now, though AI, machine learning, and neural networks, there are new frontiers to explore. The nonstop learning, the unanswered big questions, the push into the unknown - that pure adrenaline rush is back. Just when I thought I'd mapped the data world, it's drawn me into its depths once again.
Machine learning and neural networks captured my imagination back in 2015 when I attended the Data Science Workshop at CERN. In the years since, I've worked on some interesting data science projects but the innovation happening at Scale is truly groundbreaking and pushes the field forward in a major way.&nbsp;
Arun Murthy, Scale&rsquo;s Chief Product &amp; Technology Officer, has been a leader I trust since my Hortonworks days. He gave me a call to explain why he joined Scale and how pivotal this company&rsquo;s role is going to be in this new era of AI and large language models. The spark was reignited immediately. I knew this opportunity was exactly what I&rsquo;ve been looking for.
It&rsquo;s only my first month here, but based on the interactions I had during the recruitment process, the first Scale meetup in Budapest, and a few more early engagements, I&rsquo;ll say the Scaliens have something about them that is truly unique and remarkable. Everyone&rsquo;s working towards the same mission: it&rsquo;s open, efficient, relentless, collaborative, and fun. You know that you&rsquo;re at the right place and the team is on a roll, when everyone&rsquo;s working super hard yet they don&rsquo;t hesitate to crack a laugh.
At the end of the day, it&rsquo;s the technology stack, product suite, and team culture that makes a career choice sticky. All that aside, Scale is the leading AI expert company: we are pushing the envelope in autonomous driving, LLMs, computer vision, and many other arenas. I&rsquo;m thrilled to embark on this new chapter and can&rsquo;t wait to contribute to Scale&rsquo;s formative work.
For those in Budapest - I hope to catch you in one of our many meetups and fun AI events coming to &ldquo;your cinema&rdquo; soon!

Gergely joined Scale as the Regional Director of the Budapest Site t to accelerate the development of AI applications 

Why Gergely Devenyi joined Scale as the Regional Director of the Budapest Site 

Why I Joined Scale

As Scale works with governments and the leading companies around the world, one of the top questions that we are asked is, “How do we ensure that AI is safe to deploy?”&nbsp;Based on our years of work with the leading frontier model developers and enterprises deploying AI, we believe the best approach is to implement a risk-based test and evaluation framework to ensure that the AI is safe for its intended use case. For decades, test and evaluation has been a standard part of a product development cycle to ensure that products meet the necessary safety requirements to be brought to market or acquired by the government. While it may be most recognized in heavily regulated industries like aviation or medicine, test and evaluation takes place within every industry.For generative AI, such as large language models (LLMs), test and evaluation, especially red teaming, has the potential to help us understand bias in the models, weed out vulnerabilities, prevent algorithmic discrimination, and ensure that LLM developers and adopters understand the opportunities and limitations of their models.&nbsp;Today, given that LLMs are still relatively new, the methodologies and industry-wide standards to govern test and evaluation for generative AI are still being established. To help move the conversation forward, Scale recently <a href="https://static.scale.com/uploads/6019a18f03a4ae003acb1113/test-and-evaluation.pdf" rel="noopener noreferrer" target="_blank">published the industry’s first comprehensive technical methodology for scalable and effective LLM test and evaluation</a> based on the best practices we have garnered from our work across the industry.Scale’s approach features a hybrid method that combines both an automated assessment with human evaluation, resulting in a comprehensive, scalable process. While there is still much work to be done to drive industry consensus around a test and evaluation approach, and codify the standards necessary for test and evaluation of generative AI, this is a critical first step to move AI safety forward.Here are the key takeaways from Scale’s methodology:<ul><li>Comprehensive test and evaluation requires testing of both the model capabilities–defined as a model’s ability to follow instructions responsibly and factually–as well as stress testing of the safety risks associated with the model.</li><li>Scalable test and evaluation can be greatly accelerated with intelligent incorporation of state-of-the-art automated methods for both capability assessment and vulnerability detection, but will still require humans in the loop to ensure accuracy in the most complex situations. We can optimize overall evaluation efficiency without sacrificing accuracy by assessing when it is safe to trust automated approaches through computing quantitative estimates of evaluation model confidence.</li><li>A test and evaluation framework needs to be flexible enough to adapt to the risk associated with the intended use case. For example, AI for medical use cases, such as those governed by the FDA, should require more rigorous test and evaluation than AI being used for generating poetry.&nbsp;</li></ul>Read the full white paper <a href="https://static.scale.com/uploads/6019a18f03a4ae003acb1113/test-and-evaluation.pdf" rel="noopener noreferrer" target="_blank">here</a> and our vision for test and evaluation <a href="https://scale.com/guides/test-and-evaluation-vision" rel="noopener noreferrer" target="_blank">here</a>.

Based on Scale's years of work with the leading frontier model developers and enterprises deploying AI, we believe the best approach is to implement a risk-based test and evaluation framework to ensure that the AI is safe for its intended use case. 

Test & Evaluation: The Right Approach for Safe, Secure, and Trustworthy AI

Today, Scale will sign onto the Biden-Harris Administration&rsquo;s voluntary commitments to ensure that AI is safe, secure, and trustworthy. These commitments are critical to the future of AI. The reality is that progress in frontier model capabilities must happen alongside progress in model evaluation and safety. This is not only the right thing to do, but practical. &ldquo;To advance our knowledge and understanding of large language models (LLMs)&ndash;and therefore advance their capabilities&ndash;we must understand their weaknesses as much as their strengths. It is imperative for the tech industry to take proactive measures like these given the rapid pace of technological development.&rdquo; &ndash;Alexandr Wang
The Scale team commends the White House&rsquo;s leadership in bringing together companies that are shaping the future of responsible AI development. America&rsquo;s continued technological leadership hinges on our ability to build and embrace the most cutting-edge AI across all sectors of our economy and government. This can only be achieved with collaboration between industry and government to create a broad understanding of the possibilities and limitations of the technology, so the benefits can be maximized and risks minimized.&nbsp;
Safe, secure, and trustworthy AI has always been at the core of Scale&rsquo;s mission. For more than seven years, we have built deep expertise in the entire lifecycle of developing, fine-tuning, and testing and evaluating AI. Our role as the data partner for the leading model developers has given us a unique vantage point across the AI ecosystem. It has allowed us to deeply understand the challenges that must be addressed, how various players across the industry are working to tackle them, and ultimately develop our own technology to test and evaluate LLMs.&nbsp;
As a pioneer in the AI space, Scale is at the forefront of defining what it means for an LLM to be trusted and safe.
<ul>
<li>Scale is committed to helping the broader AI community understand the benefits and safety risks and partnering with leading model builders to evaluate the safety of their models. Red teaming, in particular, is a core pillar of our approach to testing and evaluation. <a href="https://www.whitehouse.gov/briefing-room/statements-releases/2023/05/04/fact-sheet-biden-harris-administration-announces-new-actions-to-promote-responsible-ai-innovation-that-protects-americans-rights-and-safety/" target="_blank" rel="noopener noreferrer">As announced by the White House&rsquo;s Office of Science and Technology Policy</a>, Scale was selected to develop the platform to evaluate leading generative AI systems for the largest red teaming exercise, to date, at DEFCON 31.&nbsp;</li>
<li>Scale published a vision for <a href="https://scale.com/guides/test-and-evaluation-vision" target="_blank" rel="noopener noreferrer">testing and evaluation </a>and developed a platform to help establish a framework for an effective industry-wide approach, limit bias, and uphold the Responsible AI Principles, such as those outlined in the <a href="https://www.whitehouse.gov/ostp/ai-bill-of-rights/" target="_blank" rel="noopener noreferrer">Biden Administration&rsquo;s AI Bill of Rights</a>. Successful adherence to and execution of these commitments is contingent on robust foundation model testing and evaluation.&nbsp;</li>
<li>Scale is grateful to join fellow industry leaders making these voluntary commitments to ensure the responsible development of AI technology. As we forge ahead, the company is committed to working with the Administration and public, private, and civil society to provide recommendations for concrete steps that can be taken to build a robust framework to unlock safe, secure, and trustworthy AI globally.</li>
</ul>

Today, Scale will sign onto the Biden-Harris Administration’s voluntary commitments to ensure that AI is safe, secure, and trustworthy.

Our Commitment to Ensuring Safe, Secure, and Trustworthy AI

Our Commitments to Safe, Secure, and Trustworthy AI

As policymakers return to Washington from the annual August recess, AI regulation will dominate political discussions. Senate Majority Leader Chuck Schumer has announced multiple &ldquo;AI Insight Forums&rdquo; beginning September 13, the White House has announced plans to issue an Executive Order, and the world is getting ready for a first-of-its-kind global AI Safety Summit in the United Kingdom this November.
Since our earliest days as a company, Scale has worked across industries to help accelerate the development of AI. Today, we work with the leading frontier model developers to fine tune, red team, and test and evaluate their large language models. This work also provides us a unique vantage point to understand, create, and implement industry best practices and understand what we believe is the right balance to strike on a regulatory approach that maximizes innovation, while putting in place the proper guardrails.
In addition to our work with the leading tech and automotive companies, Scale has worked with the U.S. government since 2020, helping to ensure the United States leads the world in AI adoption. The U.S. Department of Defense (DoD) has led the deployment of AI in the federal government to date, and there are no shortage of use cases for every agency to better enhance their day-to-day operations through AI. For example, a large language model at the U.S. Department of Energy would put information at people&rsquo;s fingertips more quickly on complex research topics like grid or hydrogen capacity for electric vehicles in major cities.&nbsp;
Our collective years of working with leading companies and the DoD has uniquely positioned Scale to understand what works and provide recommendations for the federal government to efficiently and fully embrace AI.&nbsp;
This blog is the first in our series titled, &ldquo;Scale AI Policy Framework,&rdquo; and will focus on our recommendations for successful government AI adoption based on our takeaways from our work with the DoD and broader national security community. Scale strongly believes that three things are necessary for the U.S. government to successfully adopt AI:&nbsp;
<ol>
<li>Robust AI-ready data infrastructure at each Agency&nbsp;</li>
<li>Implementation of a risk-based Test and Evaluation framework prior to AI deployment&nbsp;</li>
<li>Innovative and expedited paths to AI acquisition&nbsp;</li>
</ol>
To read the full white paper click <a href="https://pub-9ba39fa3c4764cd0bb3a9579562941d6.r2.dev/gov-ai.pdf" target="_blank" rel="noopener">here.</a>

Our first "Scale AI Policy Framework" blog focuses on successful government AI adoption, drawing from our DoD work.

Scale

Scale AI Framework for Successful Government AI Adoption

OpenAI recently announced the&nbsp;<a href="https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates" target="_blank" rel="noopener">release</a> of their GPT-3.5 Turbo fine-tuning APIs. At Scale we have always believed that building <a href="https://scale.com/custom-llms">custom LLMs</a> through fine-tuning is the key to unlocking greater performance for any given organization&rsquo;s specific use case. OpenAI also named <a href="https://scale.com/blog/open-ai-scale-partnership-gpt-3-5-fine-tuning">Scale as their preferred enterprise fine-tuning partner for GPT-3.5</a>, and we have already shown early progress for the power of fine-tuning GPT-3.5 for customers like Brex. While the pre-trained GPT-3.5 model can often solve tasks with prompt engineering, the model becomes even more powerful when fine-tuned, in some cases surpassing GPT-4 in performance. Let&rsquo;s walk you through how to fine-tune GPT-3.5 in this blog post.
The first and one of the most critical steps in fine-tuning is creating the right training dataset. OpenAI recommends creating a diverse training set of conversations that are similar to the conversations that the model will see in production. These datasets can be created with Scale&rsquo;s <a href="https://scale.com/data-engine">Data Engine</a>, which is trusted by leading ML teams and enterprises to provide large volumes of high-quality data. Scale has worked with OpenAI since 2019 on powering LLMs with better data. Scale's Data Engine has powered most of the leading LLMs, and is now also powering custom LLMs for leading companies like Brex, Chegg, and Accenture. Our cost-effective operations can give you expertly labeled and diverse data at any scale, making it an indispensable asset for fine-tuning.
Fine-tuning also includes reserving the right compute resources and properly implementing the training and inference code. OpenAI&rsquo;s new fine-tuning APIs makes this process easy and accessible. Let's dive into how you can use this API and explore some examples.
<h1 dir="ltr">Dataset Preparation</h1>
For this example, we will use <a href="https://scienceqa.github.io/" target="_blank" rel="noopener">ScienceQA</a>, a popular dataset consisting of a diverse set of multiple-choice science questions. We used <a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">LLM Engine</a> to fine-tune Llama 2 on this dataset in <a href="https://scale.com/blog/fine-tune-llama-2">our previous blogpost</a>, and you can follow the data preparation steps in that post.&nbsp;
Now, let's convert this dataset into OpenAI&rsquo;s supported format, a JSONL file with lists of conversations. Each example in the dataset should be a conversation in the same format as the chat completions endpoint, specifically a list of messages where each message has a role and content (and optionally a name). The provided assistant messages in the data should be the ideal responses you want the model to provide.
<pre class="language-python"><code>def format_chat(row): 
 return json.dumps(
 {"messages": [
 {"role": "user", "content": row["prompt"]},
 {"role": "assistant", "content": str(row["response"])},
 ]}
 )
def convert_dataset(df, file_name):
 df["conversation"] = df.apply(format_chat, axis=1),
 with open(file_name, 'w') as jsonl_file:
 for example in df["conversation"]:
 jsonl_file.write(example + '\n')</code></pre>
This results in our dataset looking like this:
<pre class="language-markup"><code>{"messages:" [{"role": "user", "content": "Context: In a group of cows, ..."}, 
 {"role": "assistant", "content": "B"}]}
["messages": {"role": "user", "content": "Context: In a group of guppies..."}, 
 {"role": "assistant", "content": "A"}]}</code></pre>
The OpenAI API supports training with a training and validation dataset, and provides loss numbers on both during the course of training. These samples will be your initial signal of how much the model is improving, when compared to generations from the base model on the same test conversations.
These dataset files must be uploaded to OpenAI&rsquo;s file endpoint. Make sure to add your OpenAI API key to your system environment variables for authentication.
<pre class="language-python"><code>import os
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
file = openai.File.create(
 file=open("scienceqa_train.jsonl", "rb"),
 purpose='fine-tune',
)</code></pre>
You should get a response similar to the following:
<pre class="language-markup"><code>{
 "object": "file",
 "id": "file-CY0FPBluqbVcoHmuGLI80lqx",
 "purpose": "fine-tune",
 "filename": "file",
 "bytes": 15974,
 "created_at": 1692743745,
 "status": "uploaded",
 "status_details": null
}</code></pre>
You can then check the status of the uploaded file using the provided file ID.&nbsp;
<pre class="language-markup"><code>openai.File.retrieve("file-CY0FPBluqbVcoHmuGLI80lqx")</code></pre>
You need to make sure that the file status is &ldquo;processed&rdquo; before proceeding with finetuning.
<h1 dir="ltr">Fine-tuning</h1>
Fine-tuning your model requires only one API call. Currently, only GPT-3.5 Turbo is available for fine-tuning. You just need to provide the IDs for the train and validation data files, the model name (only "gpt-3.5-turbo" is supported), and a suffix for the output model name.
<pre class="language-python"><code>train_data = "file-CY0FPBluqbVcoHmuGLI80lqx"
val_data = "file-428ptqQcofNSLoEP0sUDsU5x"
model = openai.FineTuningJob.create(
 model = "gpt-3.5-turbo", 
 training_file = train_data,
 validation_file = val_data,
 suffix = "scienceqa"
)</code></pre>
You would then get a response such as this:
<pre class="language-markup"><code>{
 "object": "fine_tuning.job",
 "id": "ftjob-NmNcHalWN6yGJfGnwealj7MB",
 "model": "gpt-3.5-turbo-0613",
 "created_at": 1692743811,
 "finished_at": null,
 "fine_tuned_model": null,
 "organization_id": "org-xxx",
 "result_files": [],
 "status": "created",
 "validation_file": "file-428ptqQcofNSLoEP0sUDsU5x",
 "training_file": "file-CY0FPBluqbVcoHmuGLI80lqx",
 "hyperparameters": {
 "n_epochs": 3
 },
 "trained_tokens": null
}</code></pre>
There are several ways to experiment further; for example, increasing the epochs to 4 or 5 could have different results.
You can track the progress of the training job using the finetune ID.
<pre class="language-markup"><code>openai.FineTuningJob.retrieve('ftjob-NmNcHalWN6yGJfGnwealj7MB')</code></pre>
It will have the status &ldquo;running&rdquo; while it is training and &ldquo;succeeded&rdquo; after it is finished or &ldquo;failed&rdquo; if it failed. A successful training job will look like this:
<pre class="language-markup"><code>{
 "object": "fine_tuning.job",
 "id": "ftjob-NmNcHalWN6yGJfGnwealj7MB",
 "model": "gpt-3.5-turbo-0613",
 "created_at": 1692743811,
 "finished_at": 1692744189,
 "fine_tuned_model": "ft:gpt-3.5-turbo-0613:scale-ai:test:7qUNXuwT",
 "organization_id": "org-Vn7k74eir0QoobuD5SrHfwSD",
 "result_files": ["file-hNvzZShh5Phmng1oNb1Qs4sy"],
 "status": "succeeded",
 "validation_file": "file-428ptqQcofNSLoEP0sUDsU5x",
 "training_file": "file-CY0FPBluqbVcoHmuGLI80lqx",
 "hyperparameters": {
 "n_epochs": 3
 },
 "trained_tokens": 9909
}</code></pre>
<h1 dir="ltr">Inference and Evaluation</h1>
Once you are done fine-tuning, you can view your job results. The training job will automatically upload a result file named step_metrics.csv, which contains training and test loss and accuracy for each training step.
We can use the training metrics as a rough sanity check on training stability:
<pre class="language-python"><code>from io import StringIO
def get_step_metrics(file_id):
 content = openai.File.download(file_id)
 eval_result = StringIO(result.decode())
 df = pd.read_csv(eval_result, sep=",")
 return df</code></pre>
However, inspecting the model samples gives the most relevant sense of model quality. We can run inference using our fine-tuned model using their chat completions API.
<pre class="language-python"><code>completion = openai.ChatCompletion.create(
 model="ft:gpt-3.5-turbo-0613:scale-ai:scienceqa:7qUNXuwT",
 messages=[
 {"role": "user", "content": "Context: A shopper is buying food at..."}
 ]
)</code></pre>
You will get a result like this:
<pre class="language-markup"><code>{
 "id": "chatcmpl-7qUvsuFrButlxKYSs2SispgL5KUrD",
 "object": "chat.completion",
 "created": 1692746316,
 "model": "ft:gpt-3.5-turbo-0613:scale-ai:scienceqa:7qUNXuwT",
 "choices": [
 {
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "B"
 },
 "finish_reason": "stop"
 }
 ],
 "usage": {
 "prompt_tokens": 81,
 "completion_tokens": 1,
 "total_tokens": 82
 }
}</code></pre>
<h1 dir="ltr">Results</h1>
From this fine-tuning job, the model achieves 84.58% accuracy. This is pretty good! Let's see how this compares with the original base GPT-3.5 model. Since the pre-trained model was not fine-tuned on these examples, we need to provide an example in the prompt so the model learns to adhere to the format we expect from the responses.
<div dir="ltr" align="left">
<table style="border-collapse: collapse; width: 100%; margin-left: auto; margin-right: auto;" border="1"><colgroup><col><col><col></colgroup>
<tbody>
<tr>
<td>
ScienceQA Validation
</td>
<td>
GPT-3.5
</td>
<td>
GPT-4
</td>
</tr>
<tr>
<td>
Pre-trained (1-shot)
</td>
<td>
72.98%
</td>
<td>
74.8%
</td>
</tr>
<tr>
<td>
Fine-tuned
</td>
<td>
84.58%
</td>
<td>
-
</td>
</tr>
</tbody>
</table>
</div>
&nbsp;
As we can see, fine-tuning makes a big difference in the case of all of the language models. The performance gain from fine-tuning GPT-3.5 Turbo on ScienceQA was an 11.6% absolute difference, even outperforming GPT-4!
We also experimented with different numbers of training examples. OpenAI recommends starting with 50 - 100 examples, but this can vary based on the exact use case. We can roughly estimate the expected quality gain from doubling the training data size by fine-tuning on half or a fourth of your dataset and observing the quality gap. For this experiment, we randomly sampled between 50 to 5000 examples for our training set, and used the same validation set for all models.
<div dir="ltr" align="center">
<table style="border-collapse: collapse; width: 100%; height: 275.157px; border-width: 1px;" border="1"><colgroup><col style="width: 54.321%;" width="235"><col style="width: 45.679%;" width="235"></colgroup>
<tbody>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
# Training samples
</td>
<td style="height: 47.5938px;">
Test accuracy
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
50
</td>
<td style="height: 47.5938px;">
77.18%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
100
</td>
<td style="height: 47.5938px;">
78.23%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
500
</td>
<td style="height: 47.5938px;">
81.10%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
1000
</td>
<td style="height: 47.5938px;">
81.96%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
2500
</td>
<td style="height: 47.5938px;">
84.44%
</td>
</tr>
<tr style="height: 23.5938px;">
<td style="height: 23.5938px;">
5000
</td>
<td style="height: 23.5938px;">
84.58%
</td>
</tr>
</tbody>
</table>
</div>
<img src="https://img.plasmic.app/img-optimizer/v1/img/43f22b3d99c00c724094e06687a2a7fb.png?f=webp&amp;q=75">
 We can see that performance generally increases as the number of training samples increases but plateaus at some point. However, we sampled our dataset randomly, and adding well-crafted training examples that target remaining model issues or edge cases can better improve the model&rsquo;s performance.&nbsp;
Currently, only the number of epochs is available for customization. Other hyperparameters are set to what OpenAI believes to be good default values that should largely work across a large variety of datasets and scenarios.
If you would like higher performance, OpenAI recommends collecting higher quality and/or more data. You may need to consider the balance and diversity of your current training dataset, collect more examples that target edge cases, and ensure there are no lingering issues in the existing examples. The Scale&nbsp;<a href="https://scale.com/data-engine">Data Engine</a> can deliver large volumes of high-quality data that can maximize the performance of your fine-tuned models.
<h1 dir="ltr">Conclusion</h1>
Fine-tuning GPT-3.5 makes a huge difference in unlocking greater performance from base GPT-3.5 models. Now you are all set to unleash the true potential of your fine-tuned model and witness the magic of powerful AI-generated responses. If you are looking to optimize your model performance with even more powerful features, Scale also offers an <a href="https://scale.com/custom-llms">enterprise model customization platform</a>. Happy fine-tuning!

OpenAI announced the release of their GPT-3.5 Turbo fine-tuning APIs. 

How to Fine-Tune GPT-3.5 Turbo With OpenAI API

At Scale, we host a diverse set of deep learning models for both internal and external users. The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time, we were able to reduce cost while maintaining a stable latency SLA. We&rsquo;d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with&nbsp;<a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">LLM Engine</a>.
<h2>Why Cold Start Time Is Important</h2>
Without a fast enough cold start, users will often permanently provision GPUs for peak traffic. GPU hosting costs are significantly more expensive than the average CPU-based microservice. To formalize this idea in equations,
<img style="display: block; margin-left: auto; margin-right: auto;" title="n_warm_pods.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/153d7eae-b671-4ea1-a33d-fab7dc827ae0" alt="" width="650" height="75">
If we can cold start pods and make predictions within the latency SLA, we won&rsquo;t need any warm pods; otherwise we need to keep some warm pods based on the max amount of traffic we want to keep within SLA and the throughput per node.
<img style="display: block; margin-left: auto; margin-right: auto;" title="throughput_per_node.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/4c30cc1d-9231-4298-9b0f-410d73bc3911" alt="" width="650" height="128">
Then we could calculate compute seconds per wall clock second by dividing the total amount of requests and per pod throughput and plus number of warm pods. Lastly we could get the cost for serving all requests by multiplying compute seconds with the cost per second and duration.
As a concrete example, the chart below illustrates the hosting costs of one of our products as a function of user traffic. The X-axis shows the number of requests and the Y-axis is cost. The blue line is the cost curve for a configuration that has 3x the cold start time than the red line while all other configuration values are the same. When cold start time is high (blue line), we need to keep more warm pods to maintain latency SLA when there are more requests; however if cold start time is short enough (red line), we could spin up pods right when requests come in and only keep a small amount of warm pods for safety. This difference made a huge impact on cost.
<img style="display: block; margin-left: auto; margin-right: auto;" title="line_chart.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/f92e2f99-dcf7-4516-afe6-a1707c2a11db" alt="" width="500" height="326">
<h2>Measurements</h2>
To understand where time is spent during cold start, we measured LLM endpoint cold start time for a Llama 2 7B model by looking at Kubernetes events and duration for each step that runs in the container. Here&rsquo;s the time breakdown:
<img style="display: block; margin-left: auto; margin-right: auto;" title="Cold_start_time_breakdown.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/7f1f16d9-483d-4219-b02b-04b72176513d" alt="" width="700" height="395">
As shown in the chart, the actual model loading time is quite small and most of the time is spent on pulling docker images and downloading model weights.
<h2>Faster Pod Initialization</h2>
Since pulling images from repositories takes the majority of the time, we focused our efforts here first. From our prior experiences working on optimizing inference for various deep learning models like image generation models, we&rsquo;ve already encountered similar problems, but with LLMs the docker images are bigger and the problem is more prominent. We tried a few ideas and eventually optimized away this portion of time by caching images onto the nodes using Kubernetes daemonsets. We utilized the same technique for LLM images. The following chart describes the process:
<img style="display: block; margin-left: auto; margin-right: auto;" title="llm_cold_start_flow.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/1086257a-01a0-4bbd-ad9e-04f15fcb5879" alt="" width="700" height="519">
A <a href="https://github.com/scaleapi/llm-engine/blob/main/model-engine/model_engine_server/infra/services/image_cache_service.py" target="_blank" rel="noopener">cacher</a> periodically scans through the database to get all &ldquo;<a href="https://github.com/scaleapi/llm-engine/blob/main/clients/python/llmengine/model.py#L51" target="_blank" rel="noopener">high priority</a>&rdquo; models, together with a full set of (GPU, docker image) pairs. We then construct and create/replace one daemonset for each type of GPU, and run each image with sleep commands like <code>/bin/sh -ec 'while : ; do sleep 30 ; done'</code>. This way we could dynamically maintain the set of images and preload them onto the nodes, and effectively eliminate docker image pulling time.
In addition to image caching, we reduced time to provision new nodes by creating&nbsp;<a href="https://github.com/scaleapi/llm-engine/blob/main/charts/model-engine/templates/balloon_a10_deployment.yaml" target="_blank" rel="noopener">balloon deployments</a> per GPU type to prewarm nodes. These pods have low priority and will be preempted when actual workloads are created.
<h2 dir="ltr">Faster Model Weights Loading</h2>
We utilize <a href="https://github.com/peak/s5cmd" target="_blank" rel="noopener">s5cmd</a> to download model weights which is <a href="https://joshua-robinson.medium.com/s5cmd-for-high-performance-object-storage-7071352cc09d" target="_blank" rel="noopener">much faster</a> than aws-cli. We used to put all files into a tarball for simplicity, but found that it&rsquo;s bad for concurrent download. We instead stored model files with 2GB chunks and had s5cmd download all the files in parallel. We also did some quick benchmarks for s5cmd parameters and chose 512 for --numworkers and 10 for --concurrency. With those changes we pushed the download time of the Llama 2 7B model (12.6GB) from 85s to 7s, achieving 14.4Gbps, which is close to the EBS volume bandwidth limit (16Gbps) for our host. Here is an example how we invoke s5cmd:
<pre class="language-python"><code>s5cmd --numworkers 512 cp --concurrency 10 {source_folder} {destination_folder}</code></pre>
We heavily utilize <a href="https://github.com/huggingface/text-generation-inference" target="_blank" rel="noopener">text-generation-inference</a> (TGI) to serve models. TGI prefers <a href="https://github.com/huggingface/safetensors" target="_blank" rel="noopener">safetensors</a>, a file format for storing tensors, and it loads <a href="https://huggingface.co/docs/safetensors/speed" target="_blank" rel="noopener">2 times faster</a> than raw PyTorch bin files. With safetensor files it could load the model in less than 20 seconds. Thus by default now we store only safetensors (with <code>model.save_pretrained(output_dir, safe_serialization=True</code>)) to avoid TGI having to convert PyTorch bin files on the fly.
<h2 dir="ltr">Summary</h2>
With all the optimizations applied we successfully reduced cold start time from more than 6 minutes to less than 40 seconds. This greatly improved our GPU utilization and our ability to deal with spiky traffic. Given the current 30-40s lag, it is now possible to save cost by having zero workers for sporadic workloads.
<h2 dir="ltr">Future Work</h2>
<ul>
<li dir="ltr" aria-level="1">
Though this approach conceptually works, we still need to implement (and open source) auto scaling from zero since Kubernetes&rsquo; Horizontal Pod Autoscaler does not scale from zero.
</li>
<li dir="ltr" aria-level="1">
We fine tune models with PEFT (Parameter-Efficient Fine-Tuning). The code is currently private and we will open source in the near future.
</li>
<li dir="ltr" aria-level="1">
We currently merge back PEFT trained weights onto the base model. This is cost inefficient since we need to maintain one deployment per fine-tuned model, and the cold start optimizations presented here only partially mitigate this problem. Internally, we have implemented dynamic loading / unloading for <a href="https://arxiv.org/abs/2106.09685" target="_blank" rel="noopener">LoRA</a> and <a href="https://arxiv.org/abs/2205.05638" target="_blank" rel="noopener">IA3</a> adapters with DeepSpeed-sharded models. This way only one deployment is needed per base model, and each process would dynamically swap out adapters when serving traffic from different fine-tunes. The future work is to open-source this framework and get adapters working with continuous batching.
</li>
<li dir="ltr" aria-level="1">
To better generalize our image cache beyond just &ldquo;high-priority&rdquo; endpoints, we are investigating lazy-loading of images with projects like&nbsp;<a href="https://github.com/containerd/stargz-snapshotter/tree/main" target="_blank" rel="noopener">stargz</a>. 
</li>
</ul>
&nbsp;
<h2 dir="ltr">Next Steps</h2>
Looking to customize and serve LLMs? Try&nbsp;<a href="https://github.com/scaleapi/llm-engine">LLM Engine</a> today!&nbsp;
Learn more about how Scale can help you <a href="https://scale.com/custom-llms">customize LLMs for your unique use case</a>.

The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time we were able to reduce cost, due to its significant impact on the number of warm pods if a stable latency SLA needs to be maintained. We’d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with LLM Engine.

Blog