Skip to content

Document Scoring

Document Scoring is useful when a large set of documents needs to be scored against a set of features and a scoring criteria. This usecase will give a json output for scoring for each document. Users can further rank the documents with a custom ranking criteria.

Doc Score Framework

The Doc Score Framework is designed to score/grade a set of documents against a set of features in json format. This is done by combining a prompt template and contents of the document. This framework is an extension of Prompt File Template (PFT) Framework. For further details on PFT, please refer to its documentation page.

Doc Score Data Flow

flowchart LR
    A[Documents]
    B[Features]
    C[Prompt Template]
    A --> D[DocScore Framework]
    B -->D
    C -->D
    D --> |one LLM call per document| E[LLM] --> F[LLM Response]
    F --> D
    D --> |json format|G[Scored Output Summary]
NOTE: For focused and deterministic scoring the temperature parameter of LLM are set to 0 by the framework.

In this Framework, documents are kept at user's end and sent to API one by one.

DocScoreFrameworkSingle Usecase Creation

A user can use the developer portal under the tab usecase creation or call the POST API. Follow the following steps:

  1. Prepare the prompt template, the template has to include features enclosed in {}. Here's a sample prompt in usecase testdocscore: 
    Given CV content of a candidate, and a set of features from a job description.
****
[CV]: {cv}
****
[Features]: {features}
****
[TASK]: Use the following step by step approach to evaluate the CV with the given features.
      Step 1: For each feature name, extract the relevant content from the CV.
      Step 2: Generate a remark by comparing each extracted relevant content with corresponding feature name.
      Step 3: Based on your remark, pick a score for each feature from the following list. [Very qualified, Qualified, Partially qualified, Not qualified]
      Step 4: Prepare the following details for each feature.
              - Remarks
              - Score

      Remember, only return JSON code, nothing else. It is very important.
      The following is a sample.
      {{"Remarks1": "Sample remarks", 
      "Score1": "Partially relevant",
      "Remarks2": "Sample remarks", 
      "Score2": "Not relevant"}}

In this prompt, we need to upload the cv of the candidate one by one and get the scoring.

Tips: For the prompt, it's better to be clear of the output format that you expect and give an example of the output to get better results in the desired format

  1. Fill up the rest of attributes accordingly.

Here's a sample of the header_prefix, this will be used to verify the output format returned by LLM.

header_prefix = ["remarks", "score"]
If there are two features, expected output format should be as follows: 
{{"Remarks1": "Sample remarks", 
"Score1": "Partially relevant",
"Remarks2": "Sample remarks", 
"Score2": "Not relevant"}}

Just like Prompt Template Framework, inputting multiple content for variable is also supported refer to PFT documentation page.

In the developer portal, a user can add and input multiple fields and headers accordingly.

## DocScoreFrameworkSingle Usecase Test/Debug

You can test the usecase that you've created through the frontend or debug using the developer portal under the tab Debug.

Please follow the steps below:

  1. Upload the CV of the candidate. Here's a sample, job description.

JD Game Engineer.txt: ```yaml Game Engineer

We are looking for a talented and highly motivated Game Engineer to join our team and help us deliver world-class gaming experiences. The ideal candidate will have a strong background in game development and programming, as well as a passion for creating innovative and engaging games.

Key Responsibilities:

  • Design, develop, and implement core gameplay systems, mechanics, and features.
  • Collaborate with designers, artists, and other engineers to ensure a seamless and high-quality game development process.
  • Debug and optimize game code for performance, stability, and maintainability.
  • Write clean, efficient, and well-documented code in accordance with established coding standards.
  • Participate in code and design reviews, providing and receiving constructive feedback to ensure a high standard of quality across the team.
  • Identify and solve technical challenges during development, proactively seeking solutions and anticipating potential issues.
  • Keep up-to-date with latest industry trends, technologies, and best practices in game development to continuously improve the quality of our games.

Requirements:

  • Bachelor's degree in Computer Science, Game Development, or related field, or equivalent experience.
  • 2+ years of professional experience in game development, preferably in a Game Engineer or Programmer role.
  • Strong programming skills in languages such as C++, C#, or Python.
  • Experience with game engines like Unity or Unreal Engine.
  • Familiarity with various game development platforms and tools, such as version control systems (Git, SVN), bug tracking software, and project management tools.
  • Excellent problem-solving skills and attention to detail.
  • Strong communication and teamwork skills, with the ability to collaborate effectively with teammates across disciplines.
  • Passion for gaming and a deep understanding of game design principles.

Preferred Qualifications:

  • Experience with multiplayer game development, network programming, or online services.
  • Familiarity with developing games for various platforms (PC, console, mobile).
  • Knowledge of scripting languages, such as Lua or JavaScript.
  • Experience with graphics programming, shaders, or 3D math.
  • A portfolio or demo reel showcasing previous game development projects.

What We Offer:

  • Competitive salary and benefits package.
  • A creative and collaborative work environment with opportunities for personal growth and professional development.
  • The chance to work on exciting projects with a talented and passionate team.

  • Regular team events, game nights, and other social activities. ```

  • Next, upload the features to base the scoring of the CVs on.

features.txt: sample of a list of features to score the CVs 

[
    {
      "feature_id": 1,
      "feature_name": "IT or related tertiary qualification",
    },
    {
      "feature_id": 2,
      "feature_name": "3+ years Technical Test and test automation experience",
    }
]

Here's a sample response: 

{ "Remarks1": "The candidate has a Bachelor's degree in Computer Science, Game Development, or related field, or equivalent experience, which fulfills the requirement of having an IT or related tertiary qualification.", 
"Score1": "Very qualified", 
"Remarks2": "The CV does not mention any experience in Technical Test and test automation, which is not relevant to the job description.", 
"Score2": "Not qualified" }