import os
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate1 Quick Start V1
From: Quick Start V1
1.1 LLM Chain
llm = ChatOpenAI()aimsg1 = llm.invoke("hello")aimsg1.content'Hello! How can I assist you today?'1.1.1 Simple Prompting
# Not good response
aimsg2 = llm.invoke("how can langsmith help with testing?")'Langsmith can help with testing in a few ways:\n\n1. Automated testing: Langsmith can create automated tests for your code, helping to ensure that new changes do not introduce bugs or regressions.\n\n2. Code review: Langsmith can review code changes and provide feedback on potential issues or areas for improvement.\n\n3. Performance testing: Langsmith can help to identify bottlenecks and inefficiencies in your code, allowing you to optimize performance.\n\n4. Security testing: Langsmith can help to identify potential security vulnerabilities in your code, helping to ensure that your application is secure.\n\nOverall, Langsmith can help to improve the quality and reliability of your code through testing and feedback.'print(aimsg2.content)Langsmith can help with testing in a few ways:
1. Automated testing: Langsmith can create automated tests for your code, helping to ensure that new changes do not introduce bugs or regressions.
2. Code review: Langsmith can review code changes and provide feedback on potential issues or areas for improvement.
3. Performance testing: Langsmith can help to identify bottlenecks and inefficiencies in your code, allowing you to optimize performance.
4. Security testing: Langsmith can help to identify potential security vulnerabilities in your code, helping to ensure that your application is secure.
Overall, Langsmith can help to improve the quality and reliability of your code through testing and feedback.1.1.2 Define Prompt Template
guide its response with a prompt template. Prompt templates convert raw user input to better input to the LLM.
prompt = ChatPromptTemplate.from_messages([
("system", "You are a world class technical documentation writer."),
("user", "{input}")
])promptChatPromptTemplate(input_variables=['input'], messages=[SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], template='You are a world class technical documentation writer.')), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['input'], template='{input}'))])1.1.3 Create LLM Chain
chain = prompt | llm
chainChatPromptTemplate(input_variables=['input'], messages=[SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], template='You are a world class technical documentation writer.')), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['input'], template='{input}'))])
| ChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x110e72a70>, async_client=<openai.resources.chat.completions.AsyncCompletions object at 0x110ea2530>, openai_api_key=SecretStr('**********'), openai_proxy='')aimsg3 = chain.invoke({"input": "how can langsmith help with testing?"})print(aimsg3.content)--------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In[2], line 1 ----> 1 print(aimsg3.content) NameError: name 'aimsg3' is not defined
Convert to string
from langchain_core.output_parsers import StrOutputParser
output_parser = StrOutputParser()chain = prompt | llm | output_parseraimsg4 = chain.invoke({"input": "how can langsmith help with testing?"})print(aimsg4)Langsmith can help with testing in several ways:
1. Test Automation: Langsmith can be used to automate testing processes by writing scripts to simulate user interactions with the software. This can help in running tests quickly and efficiently, saving time and effort.
2. Data Generation: Langsmith can be used to generate test data for various scenarios, helping to ensure comprehensive test coverage and uncover potential issues in the software.
3. Performance Testing: Langsmith can be used to simulate heavy loads on the software and measure its performance under stress. This can help in identifying performance bottlenecks and optimizing the software for better efficiency.
4. Regression Testing: Langsmith can be used to automate regression testing, ensuring that new updates or changes to the software do not introduce any unexpected bugs or issues.
5. Integration Testing: Langsmith can be used to automate integration testing by simulating interactions between different components of the software and verifying that they work together as expected.
Overall, Langsmith can streamline the testing process, improve test coverage, and help in delivering high-quality software to end users.1.1.4 Retrival chain
from langchain_community.document_loaders import WebBaseLoaderloader = WebBaseLoader("https://docs.smith.langchain.com/user_guide")
docs = loader.load()1.1.4.1 Embedding Model
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
embeddingsOpenAIEmbeddings(client=<openai.resources.embeddings.Embeddings object at 0x12654a170>, async_client=<openai.resources.embeddings.AsyncEmbeddings object at 0x1266392d0>, model='text-embedding-ada-002', dimensions=None, deployment='text-embedding-ada-002', openai_api_version='', openai_api_base=None, openai_api_type='', openai_proxy='', embedding_ctx_length=8191, openai_api_key=SecretStr('**********'), openai_organization=None, allowed_special=None, disallowed_special=None, chunk_size=1000, max_retries=2, request_timeout=None, headers=None, tiktoken_enabled=True, tiktoken_model_name=None, show_progress_bar=False, model_kwargs={}, skip_empty=False, default_headers=None, default_query=None, retry_min_seconds=4, retry_max_seconds=20, http_client=None, http_async_client=None, check_embedding_ctx_length=True)build our index:
from langchain_community.vectorstores import FAISS
from langchain_text_splitters import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter()
documents = text_splitter.split_documents(docs)
vector = FAISS.from_documents(documents, embeddings)documents[Document(page_content='LangSmith User Guide | \uf8ffü¶úÔ∏è\uf8ffüõ†Ô∏è LangSmith', metadata={'source': 'https://docs.smith.langchain.com/user_guide', 'title': 'LangSmith User Guide | \uf8ffü¶úÔ∏è\uf8ffüõ†Ô∏è LangSmith', 'description': 'LangSmith is a platform for LLM application development, monitoring, and testing. In this guide, we‚Äôll highlight the breadth of workflows LangSmith supports and how they fit into each stage of the application development lifecycle. We hope this will inform users how to best utilize this powerful platform or give them something to consider if they‚Äôre just starting their journey.', 'language': 'en'}),
Document(page_content='Skip to main contentLangSmith API DocsSearchGo to AppQuick StartUser GuideTracingEvaluationProduction Monitoring & AutomationsPrompt HubProxyPricingSelf-HostingCookbookThis is outdated documentation for \uf8ffü¶úÔ∏è\uf8ffüõ†Ô∏è LangSmith, which is no longer actively maintained.For up-to-date documentation, see the latest version.User GuideOn this pageLangSmith User GuideLangSmith is a platform for LLM application development, monitoring, and testing. In this guide, we‚Äôll highlight the breadth of workflows LangSmith supports and how they fit into each stage of the application development lifecycle. We hope this will inform users how to best utilize this powerful platform or give them something to consider if they‚Äôre just starting their journey.Prototyping‚ÄãPrototyping LLM applications often involves quick experimentation between prompts, model types, retrieval strategy and other parameters.\nThe ability to rapidly understand how the model is performing ‚Äî and debug where it is failing ‚Äî is incredibly important for this phase.Debugging‚ÄãWhen developing new LLM applications, we suggest having LangSmith tracing enabled by default.\nOftentimes, it isn‚Äôt necessary to look at every single trace. However, when things go wrong (an unexpected end result, infinite agent loop, slower than expected execution, higher than expected token usage), it‚Äôs extremely helpful to debug by looking through the application traces. LangSmith gives clear visibility and debugging information at each step of an LLM sequence, making it much easier to identify and root-cause issues.\nWe provide native rendering of chat messages, functions, and retrieve documents.Initial Test Set‚ÄãWhile many developers still ship an initial version of their application based on ‚Äúvibe checks‚Äù, we‚Äôve seen an increasing number of engineering teams start to adopt a more test driven approach. LangSmith allows developers to create datasets, which are collections of inputs and reference outputs, and use these to run tests on their LLM applications.\nThese test cases can be uploaded in bulk, created on the fly, or exported from application traces. LangSmith also makes it easy to run custom evaluations (both LLM and heuristic based) to score test results.Comparison View‚ÄãWhen prototyping different versions of your applications and making changes, it‚Äôs important to see whether or not you‚Äôve regressed with respect to your initial test cases.\nOftentimes, changes in the prompt, retrieval strategy, or model choice can have huge implications in responses produced by your application.\nIn order to get a sense for which variant is performing better, it‚Äôs useful to be able to view results for different configurations on the same datapoints side-by-side. We‚Äôve invested heavily in a user-friendly comparison view for test runs to track and diagnose regressions in test scores across multiple revisions of your application.Playground‚ÄãLangSmith provides a playground environment for rapid iteration and experimentation.\nThis allows you to quickly test out different prompts and models. You can open the playground from any prompt or model run in your trace.', metadata={'source': 'https://docs.smith.langchain.com/user_guide', 'title': 'LangSmith User Guide | \uf8ffü¶úÔ∏è\uf8ffüõ†Ô∏è LangSmith', 'description': 'LangSmith is a platform for LLM application development, monitoring, and testing. In this guide, we‚Äôll highlight the breadth of workflows LangSmith supports and how they fit into each stage of the application development lifecycle. We hope this will inform users how to best utilize this powerful platform or give them something to consider if they‚Äôre just starting their journey.', 'language': 'en'}),
Document(page_content="Every playground run is logged in the system and can be used to create test cases or compare with other runs.Beta Testing‚ÄãBeta testing allows developers to collect more data on how their LLM applications are performing in real-world scenarios. In this phase, it‚Äôs important to develop an understanding for the types of inputs the app is performing well or poorly on and how exactly it‚Äôs breaking down in those cases. Both feedback collection and run annotation are critical for this workflow. This will help in curation of test cases that can help track regressions/improvements and development of automatic evaluations.Capturing Feedback‚ÄãWhen launching your application to an initial set of users, it‚Äôs important to gather human feedback on the responses it‚Äôs producing. This helps draw attention to the most interesting runs and highlight edge cases that are causing problematic responses. LangSmith allows you to attach feedback scores to logged traces (oftentimes, this is hooked up to a feedback button in your app), then filter on traces that have a specific feedback tag and score. A common workflow is to filter on traces that receive a poor user feedback score, then drill down into problematic points using the detailed trace view.Annotating Traces‚ÄãLangSmith also supports sending runs to annotation queues, which allow annotators to closely inspect interesting traces and annotate them with respect to different criteria. Annotators can be PMs, engineers, or even subject matter experts. This allows users to catch regressions across important evaluation criteria.Adding Runs to a Dataset‚ÄãAs your application progresses through the beta testing phase, it's essential to continue collecting data to refine and improve its performance. LangSmith enables you to add runs as examples to datasets (from both the project page and within an annotation queue), expanding your test coverage on real-world scenarios. This is a key benefit in having your logging system and your evaluation/testing system in the same platform.Production‚ÄãClosely inspecting key data points, growing benchmarking datasets, annotating traces, and drilling down into important data in trace view are workflows you‚Äôll also want to do once your app hits production.However, especially at the production stage, it‚Äôs crucial to get a high-level overview of application performance with respect to latency, cost, and feedback scores. This ensures that it's delivering desirable results at scale.Online evaluations and automations allow you to process and score production traces in near real-time.Additionally, threads provide a seamless way to group traces from a single conversation, making it easier to track the performance of your application across multiple turns.Monitoring and A/B Testing‚ÄãLangSmith provides monitoring charts that allow you to track key metrics over time. You can expand to view metrics for a given period and drill down into a specific data point to get a trace table for that time period ‚Äî this is especially handy for debugging production issues.LangSmith also allows for tag and metadata grouping, which allows users to mark different versions of their applications with different identifiers and view how they are performing side-by-side within each chart. This is helpful for A/B testing changes in prompt, model, or retrieval strategy.Automations‚ÄãAutomations are a powerful feature in LangSmith that allow you to perform actions on traces in near real-time. This can be used to automatically score traces, send them to annotation queues, or send them to datasets.To define an automation, simply provide a filter condition, a sampling rate, and an action to perform. Automations are particularly helpful for processing traces at production scale.Threads‚ÄãMany LLM applications are multi-turn, meaning that they involve a series of interactions between the user and the application. LangSmith provides a threads view that groups traces from a single conversation", metadata={'source': 'https://docs.smith.langchain.com/user_guide', 'title': 'LangSmith User Guide | \uf8ffü¶úÔ∏è\uf8ffüõ†Ô∏è LangSmith', 'description': 'LangSmith is a platform for LLM application development, monitoring, and testing. In this guide, we‚Äôll highlight the breadth of workflows LangSmith supports and how they fit into each stage of the application development lifecycle. We hope this will inform users how to best utilize this powerful platform or give them something to consider if they‚Äôre just starting their journey.', 'language': 'en'}),
Document(page_content='applications are multi-turn, meaning that they involve a series of interactions between the user and the application. LangSmith provides a threads view that groups traces from a single conversation together, making it easier to track the performance of and annotate your application across multiple turns.Was this page helpful?You can leave detailed feedback on GitHub.PreviousQuick StartNextOverviewPrototypingBeta TestingProductionCommunityDiscordTwitterGitHubDocs CodeLangSmith SDKPythonJS/TSMoreHomepageBlogLangChain Python DocsLangChain JS/TS DocsCopyright ¬© 2024 LangChain, Inc.', metadata={'source': 'https://docs.smith.langchain.com/user_guide', 'title': 'LangSmith User Guide | \uf8ffü¶úÔ∏è\uf8ffüõ†Ô∏è LangSmith', 'description': 'LangSmith is a platform for LLM application development, monitoring, and testing. In this guide, we‚Äôll highlight the breadth of workflows LangSmith supports and how they fit into each stage of the application development lifecycle. We hope this will inform users how to best utilize this powerful platform or give them something to consider if they‚Äôre just starting their journey.', 'language': 'en'})]1.1.4.2 Create Retrival Chain
A retrieval chain: this chain will take an incoming question, look up relevant documents, then pass those documents along with the original question into an LLM and ask it to answer the original question.
from langchain.chains.combine_documents import create_stuff_documents_chain
prompt = ChatPromptTemplate.from_template("""Answer the following question based only on the provided context:
<context>
{context}
</context>
Question: {input}""")
document_chain = create_stuff_documents_chain(llm, prompt)
document_chainRunnableBinding(bound=RunnableBinding(bound=RunnableAssign(mapper={
context: RunnableLambda(format_docs)
}), config={'run_name': 'format_inputs'})
| ChatPromptTemplate(input_variables=['context', 'input'], messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'input'], template='Answer the following question based only on the provided context:\n\n<context>\n{context}\n</context>\n\nQuestion: {input}'))])
| ChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x13ff3d9f0>, async_client=<openai.resources.chat.completions.AsyncCompletions object at 0x13ff3f7c0>, openai_api_key=SecretStr('**********'), openai_proxy='')
| StrOutputParser(), config={'run_name': 'stuff_documents_chain'})Run this ourselves by passing in documents directly:
from langchain_core.documents import Document
aimsg5 = document_chain.invoke({
"input": "how can langsmith help with testing?",
"context": [Document(page_content="langsmith can let you visualize test results")]
})aimsg5'Langsmith can help with testing by allowing you to visualize test results.'From Full Document
aimsg6 = document_chain.invoke({
"input": "how can langsmith help with testing?",
"context": documents
})print(aimsg6)LangSmith can help with testing by allowing developers to create datasets, run tests on LLM applications using these datasets, upload test cases in bulk or create them on the fly, export test cases from application traces, and run custom evaluations to score test results. Additionally, LangSmith provides a comparison view for tracking and diagnosing regressions in test scores across multiple revisions of an application. The platform also offers a playground environment for rapid iteration and experimentation, logging every playground run for creating test cases or comparing with other runs. Finally, LangSmith supports beta testing to collect more data on how LLM applications perform in real-world scenarios, capture feedback from users, and annotate traces for evaluation criteria.However, we want the documents to first come from the retriever we just set up. That way, we can use the retriever to dynamically select the most relevant documents and pass those in for a given question.
from langchain.chains import create_retrieval_chain
retriever = vector.as_retriever()
retrieval_chain = create_retrieval_chain(retriever, document_chain)Invoke this chain - This returns a dictionary - the response from the LLM is in the answer key
aimsg7 = retrieval_chain.invoke({"input": "how can langsmith help with testing?"})
print(aimsg7["answer"])LangSmith can help with testing by allowing developers to create datasets, run tests on LLM applications using these datasets, and easily evaluate the results. Developers can upload test cases in bulk, create them on the fly, or export them from application traces. LangSmith also supports custom evaluations (LLM and heuristic based) to score test results. Additionally, LangSmith provides a comparison view for tracking and diagnosing regressions in test scores across multiple revisions of an application. The platform also offers a playground environment for rapid iteration and experimentation, allowing users to test out different prompts and models and log each run for further analysis.