Class CriteriaEvalChain

Hierarchy (view full)

LLMStringEvaluator
- CriteriaEvalChain
  - LabeledCriteriaEvalChain

Constructors

constructor

new CriteriaEvalChain(fields): CriteriaEvalChain
Parameters
- fields: LLMChainInput<EvalOutputType, BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>>
Returns CriteriaEvalChain
Inherited from LLMStringEvaluator.constructor
- Defined in langchain/src/chains/llm_chain.ts:123

Properties

llm

llm: BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>

outputKey

outputKey: string = "text"

outputParser

outputParser: BaseLLMOutputParser<EvalOutputType> = ...

prompt

prompt: BasePromptTemplate<any, BasePromptValueInterface, any>

requiresInput

requiresInput: boolean = true

requiresReference

requiresReference: boolean = false

skipReferenceWarning

skipReferenceWarning: string = ...

`Optional` criterionName

criterionName?: string

`Optional` evaluationName

evaluationName?: string = ...

`Optional` llmKwargs

llmKwargs?: any

`Optional` memory

memory?: BaseMemory

`Optional` skipInputWarning

skipInputWarning?: string = ...

Accessors

inputKeys

get inputKeys(): string[]
Returns string[]
Inherited from LLMStringEvaluator.inputKeys
- Defined in langchain/src/chains/llm_chain.ts:115

outputKeys

get outputKeys(): string[]
Returns string[]
Inherited from LLMStringEvaluator.outputKeys
- Defined in langchain/src/chains/llm_chain.ts:119

Methods

apply

apply(inputs, config?): Promise<ChainValues[]>
Parameters
- inputs: ChainValues[]
- Optional config: (RunnableConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[])[]
Returns Promise<ChainValues[]>

⚠️ Deprecated ⚠️

Use .batch() instead. Will be removed in 0.2.0.

This feature is deprecated and will be removed in the future.

It is not recommended for use.

Call the chain on all inputs in the list
Inherited from LLMStringEvaluator.apply
- Defined in langchain/src/chains/base.ts:258

call

call(values, config?): Promise<ChainValues>
Run the core logic of this chain and add to output if desired.

Wraps _call and handles memory.
Parameters
- values: any
- Optional config: BaseCallbackConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]
Returns Promise<ChainValues>
Inherited from LLMStringEvaluator.call
- Defined in langchain/src/chains/llm_chain.ts:180

checkEvaluationArgs

checkEvaluationArgs(reference?, input?): void
Check if the evaluation arguments are valid.
Parameters
- Optional reference: string
  The reference label.
- Optional input: string
  The input string.
Returns void
Throws
If the evaluator requires an input string but none is provided, or if the evaluator requires a reference label but none is provided.
Inherited from LLMStringEvaluator.checkEvaluationArgs
- Defined in langchain/src/evaluation/base.ts:62

evaluateStrings

evaluateStrings(args, config?): Promise<ChainValues>
Evaluate Chain or LLM output, based on optional input and label.
Parameters
- args: StringEvaluatorArgs & BaseLanguageModelCallOptions
- Optional config: BaseCallbackConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]
Returns Promise<ChainValues>
The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys:
- score: the score of the evaluation, if applicable.
- value: the string value of the evaluation, if applicable.
- reasoning: the reasoning for the evaluation, if applicable.
Inherited from LLMStringEvaluator.evaluateStrings
- Defined in langchain/src/evaluation/base.ts:196

getEvalInput

getEvalInput(__namedParameters): CriteriaEvalInput
Parameters
- __namedParameters: StringEvaluatorArgs
Returns CriteriaEvalInput
- Defined in langchain/src/evaluation/criteria/criteria.ts:245

invoke

invoke(input, options?): Promise<ChainValues>
Invoke the chain with the provided input and returns the output.
Parameters
- input: ChainValues
  Input values for the chain run.
- Optional options: RunnableConfig
Returns Promise<ChainValues>
Promise that resolves with the output of the chain run.
Inherited from LLMStringEvaluator.invoke
- Defined in langchain/src/chains/base.ts:86

predict

predict(values, callbackManager?): Promise<EvalOutputType>
Format prompt with values and pass to LLM
Parameters
- values: any
  keys to pass to prompt template
- Optional callbackManager: CallbackManager
  CallbackManager to use
Returns Promise<EvalOutputType>
Completion from LLM.
Example
```
llm.predict({ adjective: "funny" })
```
Inherited from LLMStringEvaluator.predict
- Defined in langchain/src/chains/llm_chain.ts:246

prepOutputs

prepOutputs(inputs, outputs, returnOnlyOutputs?): Promise<Record<string, unknown>>
Parameters
- inputs: Record<string, unknown>
- outputs: Record<string, unknown>
- returnOnlyOutputs: boolean = false
Returns Promise<Record<string, unknown>>
Inherited from LLMStringEvaluator.prepOutputs
- Defined in langchain/src/chains/base.ts:149

run

run(input, config?): Promise<string>
Parameters
- input: any
- Optional config: RunnableConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]
Returns Promise<string>
Deprecated
Use .invoke() instead. Will be removed in 0.2.0.
Inherited from LLMStringEvaluator.run
- Defined in langchain/src/chains/base.ts:190

serialize

serialize(): SerializedLLMChain
Returns SerializedLLMChain
Deprecated
Inherited from LLMStringEvaluator.serialize
- Defined in langchain/src/chains/llm_chain.ts:274

`Static` deserialize

deserialize(data): Promise<LLMChain<string, BaseLanguageModel<any, BaseLanguageModelCallOptions>>>
Parameters
- data: SerializedLLMChain
Returns Promise<LLMChain<string, BaseLanguageModel<any, BaseLanguageModelCallOptions>>>
Inherited from LLMStringEvaluator.deserialize
- Defined in langchain/src/chains/llm_chain.ts:258

`Static` fromLLM

fromLLM(llm, criteria?, chainOptions?): Promise<CriteriaEvalChain>
Create a new instance of the CriteriaEvalChain.
Parameters
- llm: BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>
- Optional criteria: "detail" | ConstitutionalPrinciple | {
  [key: string]: string;
  } | "conciseness" | "relevance" | "correctness" | "coherence" | "harmfulness" | "maliciousness" | "helpfulness" | "controversiality" | "misogyny" | "criminality" | "insensitivity" | "depth" | "creativity"
- Optional chainOptions: Partial<Omit<LLMEvalChainInput<EvalOutputType, BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>>, "llm">>
  Options to pass to the constructor of the LLMChain.
Returns Promise<CriteriaEvalChain>
- Defined in langchain/src/evaluation/criteria/criteria.ts:209

`Static` resolveCriteria

resolveCriteria(criteria?): Record<string, string>
Resolve the criteria to evaluate.
Parameters
- Optional criteria: "detail" | ConstitutionalPrinciple | {
  [key: string]: string;
  } | "conciseness" | "relevance" | "correctness" | "coherence" | "harmfulness" | "maliciousness" | "helpfulness" | "controversiality" | "misogyny" | "criminality" | "insensitivity" | "depth" | "creativity"
  The criteria to evaluate the runs against. It can be: - a mapping of a criterion name to its description - a single criterion name present in one of the default criteria - a single ConstitutionalPrinciple instance
Returns Record<string, string>
A dictionary mapping criterion names to descriptions.
- Defined in langchain/src/evaluation/criteria/criteria.ts:150

`Static` resolvePrompt

resolvePrompt(prompt?): BasePromptTemplate<any, BasePromptValueInterface, any>
Resolve the prompt to use for the evaluation.
Parameters
- Optional prompt: BasePromptTemplate<any, BasePromptValueInterface, any>
Returns BasePromptTemplate<any, BasePromptValueInterface, any>
- Defined in langchain/src/evaluation/criteria/criteria.ts:183

Class CriteriaEvalChain

Hierarchy (view full)

Index

Constructors

Properties

Accessors

Methods

Constructors

constructor

Parameters

fields: LLMChainInput<EvalOutputType, BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>>

Returns CriteriaEvalChain

Properties

llm

outputKey

outputParser

prompt

requiresInput

requiresReference

skipReferenceWarning

Optional criterionName

Optional evaluationName

Optional llmKwargs

Optional memory

Optional skipInputWarning

Accessors

inputKeys

Returns string[]

outputKeys

Returns string[]

Methods

apply

Parameters

inputs: ChainValues[]

Optional config: (RunnableConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[])[]

Returns Promise<ChainValues[]>

⚠️ Deprecated ⚠️

call

Parameters

values: any

Optional config: BaseCallbackConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]

Returns Promise<ChainValues>

checkEvaluationArgs

Parameters

Optional reference: string

Optional input: string

Returns void

Throws

evaluateStrings

Parameters

args: StringEvaluatorArgs & BaseLanguageModelCallOptions

Optional config: BaseCallbackConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]

Returns Promise<ChainValues>

getEvalInput

Parameters

__namedParameters: StringEvaluatorArgs

Returns CriteriaEvalInput

invoke

Parameters

input: ChainValues

Optional options: RunnableConfig

Returns Promise<ChainValues>

predict

Parameters

values: any

Optional callbackManager: CallbackManager

Returns Promise<EvalOutputType>

Example

prepOutputs

Parameters

inputs: Record<string, unknown>

outputs: Record<string, unknown>

returnOnlyOutputs: boolean = false

Returns Promise<Record<string, unknown>>

run

Parameters

input: any

Optional config: RunnableConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]

Returns Promise<string>

Deprecated

`Optional` criterionName

`Optional` evaluationName

`Optional` llmKwargs

`Optional` memory

`Optional` skipInputWarning

`Optional` config: (RunnableConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[])[]

`Optional` config: BaseCallbackConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]

`Optional` reference: string

`Optional` input: string

`Optional` config: BaseCallbackConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]

`Optional` options: RunnableConfig

`Optional` callbackManager: CallbackManager

`Optional` config: RunnableConfig | CallbackManager | (BaseCallbackHandler | BaseCallbackHandlerMethodsClass)[]

`Static` deserialize

`Static` fromLLM

`Optional` criteria: "detail" | ConstitutionalPrinciple | {
[key: string]: string;
} | "conciseness" | "relevance" | "correctness" | "coherence" | "harmfulness" | "maliciousness" | "helpfulness" | "controversiality" | "misogyny" | "criminality" | "insensitivity" | "depth" | "creativity"

`Optional` chainOptions: Partial<Omit<LLMEvalChainInput<EvalOutputType, BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>>, "llm">>

`Static` resolveCriteria

`Optional` criteria: "detail" | ConstitutionalPrinciple | {
[key: string]: string;
} | "conciseness" | "relevance" | "correctness" | "coherence" | "harmfulness" | "maliciousness" | "helpfulness" | "controversiality" | "misogyny" | "criminality" | "insensitivity" | "depth" | "creativity"

`Static` resolvePrompt

`Optional` prompt: BasePromptTemplate<any, BasePromptValueInterface, any>