How BQA streamlines education quality reporting using Amazon Bedrock

Given the amount of data today, organizations across a variety of industries work with large amounts of data in multiple formats. Manually reviewing and processing this information can be a challenging and time-consuming task, with a margin of error. This is where intelligent document processing (IDP), coupled with the power of generative AI, emerges as a game-changing solution.

Enhancing IDP capabilities is the integration of generative AI, which uses large-scale linguistic models (LLMs) and generative techniques to understand and generate human-like text. This integration allows organizations to not only extract data from documents, but also translate, summarize, and generate insights from the extracted information, enabling a more intelligent and automated document processing workflow.

The Education and Training Quality Authority (BQA) plays an important role in improving the quality of education and training services in the Kingdom of Bahrain. BQA reviews the performance of all educational and training institutions, including schools, universities, and vocational institutes, thereby promoting the professional development of the nation's workforce.

BQA oversees the overall quality assurance process, which includes setting performance standards and conducting targeted reviews of education and training institutions. This process involves the collection and analysis of extensive documents, including self-evaluation reports (SERs), supporting evidence, and various types of media from the institutions being reviewed.

The collaboration between BQA and AWS was done through the Cloud Innovation Center (CIC) program, which is a joint initiative between AWS, Tamkeen, and leading universities in Bahrain, including Bahrain Polytechnic and the University of Bahrain. The CIC program aims to stimulate innovation in the public sector by providing a collaborative environment where government agencies can work closely with AWS consultants and university students to develop cutting-edge solutions using the latest cloud technologies.

As part of the CIC program, BQA built a proof of concept solution, leveraging the power of AWS services and AI productivity capabilities. The primary purpose of this proof of concept was to test and validate the proposed technology, demonstrating its functionality and potential to simplify BQA reporting and data management processes.

In this post, we explore how BQA leveraged the power of Amazon Bedrock, Amazon SageMaker JumpStart, and other AWS services to streamline the overall reporting workflow.

Challenge: Simplifying self-assessment reporting

BQA often provides education and training institutions with a SER template as part of the review process. Institutions are required to submit a review portfolio containing the completed SER and supporting materials as evidence, which sometimes did not fully comply with established reporting standards.

The existing process had some challenges:

• Incorrect or incomplete proposals – Institutions may provide incomplete or inaccurate information in submitted reports and supporting evidence, leading to gaps in the data needed for a comprehensive review.
• Absent or insufficient supporting evidence – The supporting material provided as evidence by the institutions often did not confirm the claims made in their reports, challenging the evaluation process.
• It is time consuming and resource intensive – The process required devoting significant time and resources to review submissions manually and follow up with agencies to request additional information if needed to prepare submissions, leading to delays in the overall review process.

These challenges have highlighted the need for a more systematic and effective approach to delivery and review.

Solution overview

The proposed solution uses Amazon Bedrock and the Amazon Titan Express model to enable IDP functionality. The architecture seamlessly integrates multiple AWS services with Amazon Bedrock, enabling efficient data extraction and comparison.

Amazon Bedrock is a fully managed service that provides access to the most effective foundational models (FMs) from leading AI startups and Amazon through a unified API. It offers a wide range of FMs, allowing you to choose the model that best suits your specific use case.

The following diagram shows the structure of the solution.

The solution consists of the following steps:

1. Relevant documents are uploaded and stored in an Amazon Simple Storage Service (Amazon S3) bucket.
2. The event notification is sent to the Amazon Simple Queue Service (Amazon SQS) queue to queue each file for further processing. Amazon SQS acts as a buffer, allowing different components to send and receive messages reliably without direct integration, improving the robustness and fault tolerance of the system.
3. The AWS Lambda text extraction function is invoked by the SQS queue, processes each file in the queue and uses Amazon Textract to extract the text from the documents.
4. The extracted text data is entered into another SQS queue for the next processing step.
5. The Lambda text summarization function is invoked by this newline containing the extracted text. This function sends a request to SageMaker JumpStart, where the Meta Llama text generation model is used to summarize content based on the information provided.
6. Correspondingly, the InvokeSageMaker Lambda function is invoked to perform comparisons and checks. It compares the extracted text against the BQA standards the model was trained on, analyzing the text for relevance, quality, and other relevant metrics.
7. Summary data and test results are stored in an Amazon DynamoDB table
8. On request, the InvokeBedrock Lambda function asks Amazon Bedrock to generate productive AI summaries and comments. The project created detailed information designed to guide the Amazon Titan Express model in university delivery tests.

Rapid engineering using Amazon Bedrock

In order to utilize the power of Amazon Bedrock and ensure that the output meets the desired structure and formatting requirements, carefully designed information was developed according to the following guidelines:

• Submission of evidence – Present the evidence submitted by the institution under the appropriate reference, providing the model and context required for the examination
• Test criteria – Define the specific criteria by which evidence should be evaluated
• Test instructions – Order the model as follows:
  • Show that it does not exist if the proof is not related to the index
  • Evaluate the university's self-evaluation based on the principles
  • Give points from 1–5 to each comment, citing evidence from the content
• Response format – Explain the answer as bullet points, focusing on relevant analysis and evidence, with a word limit of 100 words

To use this notification template, you can create a custom Lambda function for your project. The employee must handle the retrieval of the necessary data, such as the name of the reference, the evidence submitted by the university, and the rubric process. Inside the function, enter the information template and dynamically fill in the placeholders (${indicatorName}, ${JSON.stringify(allContent)}again ${JSON.stringify(c.comment)}) with the data returned.

The Amazon Titan Text Express model will then generate a test response based on the given information instructions, adhering to the specified format and guidelines. You can process and analyze the model's feedback within your work, extracting relevance scores, relevant analysis, and evidence.

The following is a sample information template:

for (const c of comments) {
        const prompt = `
        Below is the evidence submitted by the university under the indicator "${indicatorName}":
        ${JSON.stringify(allContent)}
    
         Analyze and Evaluate the university's eviedence based on the provided rubric criteria:
        ${JSON.stringify(c.comment)}

        - If the evidence does not relate to the indicator, indicate that it is not applicable (N/A) without any additional commentary.
        
       Choose one from the below compliance score based on evidence submitted:
       1. Non-compliant: The comment does not meet the criteria or standards.
        2.Compliant with recommendation: The comment meets the criteria but includes a suggestion or recommendation for improvement.
        3. Compliant: The comment meets the criteria or standards.

        THE END OF THE RESPONSE THERE SHOULD BE EITHER SCORE: [SCORE: COMPLIANT OR NON-COMPLIANT OR COMPLIANT WITH RECOMMENDATION]
        Write your response in concise bullet points, focusing strictly on relevant analysis and evidence.
        **LIMIT YOUR RESPONSE TO 100 WORDS ONLY.**

        `;

        logger.info(`Prompt for comment ${c.commentId}: ${prompt}`);

        const body = JSON.stringify({
          inputText: prompt,
          textGenerationConfig: {
            maxTokenCount: 4096,
            stopSequences: [],
            temperature: 0,
            topP: 0.1,
          },
        });

The following screenshot shows an example of the response generated by Amazon Bedrock.

Results

The implementation of Amazon Bedrock has given institutions revolutionary benefits. By automating and simplifying the collection and analysis of extensive documents, including SERs, supporting evidence, and various media formats, institutions can achieve greater accuracy and consistency in their reporting processes and readiness for the review process. This not only reduces the time and costs associated with manual data processing, but also improves compliance with quality expectations, thus improving the reliability and quality of their institutions.

For BQA the implementation helped achieve one of its strategic goals focused on simplifying their reporting processes and achieving significant improvements across a range of key metrics, greatly improving the efficiency and effectiveness of their operations.

Expected key success metrics include:

• Faster turnaround times for producing 70% accurate and standards-compliant self-test reports, resulting in improved overall performance.
• Reduced risk of errors or non-compliance with the reporting process, which enforces compliance with established guidelines.
• The ability to summarize long submissions into short points, allowing BQA reviewers to quickly analyze and understand the most relevant information, reducing evidence analysis time by 30%.
• More accurate compliance feedback functionality, empowering reviewers to effectively analyze shipments against established standards and guidelines, while achieving 30% reduced operational costs through process improvements.
• Improved transparency and communication through seamless interactions, allowing users to easily request additional documents or explanations.
• Real-time feedback, allowing institutions to make the necessary changes quickly. This is especially useful to maintain the accuracy and completeness of submissions.
• Improved decision making by providing insights with data. This helps universities identify areas for improvement and make data-driven decisions to improve their processes and operations.

The following screenshot shows an example of creating a new experiment using Amazon Bedrock

The conclusion

This post featured Amazon Bedrock's implementation at the Education and Training Quality Authority (BQA), demonstrating the transformative power of productive AI in transforming quality assurance processes in the education and training sectors. For those interested in exploring the technical details further, the full code for this implementation is available in the following GitHub repo. If you would like to do a similar proof of concept with us, submit your challenge idea on the Bahrain Polytechnic or University of Bahrain CIC website.

About the Author

Maram AlSaegh is a Cloud Infrastructure Architect at Amazon Web Services (AWS), where he supports AWS customers in accelerating their journey to the cloud. Currently, he is focused on developing innovative solutions that leverage productive AI and machine learning (ML) for public sector companies.