I am working on a regulatory intelligence product. We are using LLMs in multiple document processing tasks. One of the tasks that I was working on today required me to extract text from a PDF document. The PDF in question is a circular from SAMA (The Saudi Central Bank). The 2-page long PDF is in Arabic language and looks like a natively digital document with consistent formatting rather than a scanned document.
The task that I wanted to perform was to extract the Arabic text and translate it to English. This looks like an easy task that most LLMs should be able to do since they all support PDF processing and they can read and write different languages.
Below is the screenshot of the first page of the document.

I tried four models on this PDF: gpt-4o via https://chatgpt.com/, Grok 3 via https://grok.com/, Gemini 2.0 Flash via https://gemini.google.com/, and Claude 3.7 Sonnet via https://claude.ai/. Below are my findings.
For all the models, I just uploaded the PDF and prompted Extract text as English.
1. gpt-4o via ChatGPT
ChatGPT failed to upload this file. I tried other files and it worked, but for this file it always gave an Unknown error occurred message. Interestingly, there was no error in the console or network tab. I think it processes the file fine but when it is trying to show a preview of the file, ChatGPT fails.
2. Grok 3 via grok.com
I wrote a blog last month where I mentioned that Grok 3 with thinking is impressive. I have used Grok 3 for coding-related tasks but this was the first time I tried it for a PDF processing task. Grok 3 failed badly. It extracted some text but did a poor job.
Below is the extracted text when I asked Grok 3 to extract the text as English.
The document you provided contains text extracted via OCR, with two pages. Here's the extracted text translated or interpreted into English where applicable:
---
### PAGE 1
```
46043115
1446/08/18
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
معمد
```
**Interpretation:**
- `46043115`: Likely a reference number or ID.
- `1446/08/18`: This appears to be a date in the Islamic (Hijri) calendar, which corresponds to approximately March 6, 2025, in the Gregorian calendar (based on the current date of March 4, 2025, provided in the context).
- `معمد`: This Arabic word is repeated multiple times. It translates to "Mohammed" (a common name) in English. However, the repetition suggests it might be an OCR error or placeholder text rather than meaningful content.
---
### PAGE 2
```
Saudi Central Bank
[Rest of the content appears to be blank or unreadable via OCR]
```
**Interpretation:**
- `Saudi Central Bank`: This is the only clear English text on the second page, indicating that the document may be related to an official communication or form from the Saudi Central Bank (also known as SAMA - Saudi Arabian Monetary Authority).
- The remaining content appears to be empty or not properly captured by OCR, possibly due to formatting, faint text, or an incomplete scan.
---
### Summary of Extracted English Text
- **Page 1:**
- `46043115`
- `1446/08/18` (Hijri date, approx. March 6, 2025)
- Repeated text `معمد` ("Mohammed" in Arabic, possibly an error or filler).
- **Page 2:**
- `Saudi Central Bank`
If you need further analysis or have a specific question about this document, please let me know!
- It fails to extract the complete text. In the extracted text it mentions
[Rest of the content appears to be blank or unreadable via OCR] - Grok 3 vision capabilities are poor
- For some reason it tries to interpret and summarize the content, but since it failed to extract text, it was not able to do much
3. Gemini 2.0 Flash via Gemini web app
I had high hopes for Gemini. I have read multiple posts on HackerNews in the last month where authors have spoken highly of Gemini’s PDF processing capabilities. So, it looked like a simple task for Gemini.
When I prompted Gemini to Extract text as English, it gave the following output.
I am sorry, but I cannot fulfill this request. The provided text is already in English, so there is no need to extract it as English again.
I made a follow-up message: I meant text in the PDF document. It is in Arabic. Then, it gave me a short summary.
This document is an official notice related to the readiness of Saudi Central Bank (SAMA)currency exchange center (SAMA) for the Hajj and Ramadan seasons of 1446. It discusses the readiness of exchange center locations and the operation of full capacity of exchange centers during Ramadan and Hajj season of 1446 AH.
Next, I asked that it should give me Extract the complete text not a summary, but it still gave me a short summary.
This document is an official notice related to the readiness of Saudi Central Bank (SAMA)currency exchange center (SAMA) for the Hajj and Ramadan seasons of 1446. It discusses the readiness of exchange center locations and the operation of full capacity of exchange centers during Ramadan and Hajj season of 1446 AH.
4. Claude 3.7 Sonnet via Claude.ai
I use Claude 3.7 Sonnet with Extended thinking in all my coding-related tasks. I use Claude often for PDF processing and text extraction tasks as well. Since I had a bad experience with all the models so far, I was not sure what to expect from Claude 3.7 Sonnet. Claude 3.7 Sonnet worked flawlessly. Below is the text it extracted.
This is a circular from the Saudi Central Bank regarding the readiness of currency exchange centers during Ramadan and Hajj seasons. Here's a translation of the content:
# Saudi Central Bank Circular
**Number:** 46043115
**Date:** 1446/08/18
**Attachments:** 1 attachment
## Circular
**To:** The Respected Individuals
**Peace be upon you, and God's mercy and blessings.**
**Subject:** Confirmation of the readiness of facilities and full operational capacity of exchange centers during Ramadan and Hajj seasons for the year 1446H.
With reference to paragraph (B) of Article 14 of the regulations governing currency exchange operations, which states that "the Central Bank may issue any instructions it deems necessary to apply supervisory and regulatory requirements on the operations of exchange centers," and based on the main objectives of the Pilgrims Service Program - one of the Kingdom's Vision 2030 programs - regarding providing high-quality services to pilgrims and visitors, achieving an 85% satisfaction rate with services provided to pilgrims and visitors by 2025, and 90% by 2030, and the importance of the exchange sector as one of the services that contribute to achieving this goal.
Therefore, the Central Bank emphasizes to all exchange centers located in Makkah, Madinah, King Abdulaziz International Airport, and the Pilgrims City in Jeddah Governorate the importance of raising their level of readiness and early preparation to provide foreign currency purchase and sale services effectively within the Kingdom during Ramadan and Hajj seasons, to achieve the aforementioned targets, while complying with all regulations and instructions, including but not limited to those detailed in Attachment No. (1).
For your information and action accordingly.
Regards,
**Yazid bin Ahmed Al-Sheikh**
**Deputy Governor for Supervision**
---
### Attachment No. (1)
1. Compliance with the regulations governing currency exchange operations and related circulars.
2. Full operational capacity through activation of all exchange windows during the season.
3. Providing sufficient cash and all Saudi Riyal denominations in sufficient time to ensure service provision that matches the needs of all visitors and pilgrims during the season and during Eid al-Fitr and Eid al-Adha.
4. Extending center working hours and days appropriately during Ramadan, Hajj, and the Eid holidays.
5. Preventing customer crowding inside and outside the branch by taking the following minimum procedures (with dedicated staff to monitor and ensure this):
- Providing and activating customer queue management systems
- Providing and activating customer queue management devices
- Providing waiting chairs for customers
6. Commitment to displaying staff ID badges with names, roles, and work hours during the season.
7. Displaying currency exchange rates in a prominent location.
8. Updating the currency exchange rate display screen directly through the system.
9. Issuing receipts for all transactions and providing a copy to the customer.
10. Placing clear signs and notices indicating customer rights.
11. Making point-of-sale devices available, in addition to providing electronic payment service for customers wishing to pay through it.
12. Ensuring surveillance cameras and monitoring units are functioning effectively.
13. Providing sufficient security guards.
14. Providing modern equipment for cash counting and counterfeit currency detection along with following necessary procedures when dealing with counterfeit currency.
15. Adding the geographical location of the center on Google Maps, in addition to writing the center's name in both Arabic and English.
16. Commitment to notifying security authorities in cases of detecting currency exchange operations from unauthorized outlets, while informing the Central Bank of such cases.
There is an English version of the document https://rulebook.sama.gov.sa/en/emphasizing-readiness-headquarters-and-full-operational-capacity-exchange-centers-during-ramadan in HTML format, so I compared the text against it to see if there were any mistakes. Apart from minor language differences, the extracted text is consistent with the original English HTML document.
Claude 3.7 Sonnet keeps impressing me more and more. I am happy to pay its monthly $20 subscription.
5. Mistral OCR
Mistral released their OCR offering today(7th March 2025) so I gave it a try as well. Mistral OCR made multiple mistakes in extracting this document. It is a two-page-long PDF in Arabic from the Saudi Central Bank. The following errors were observed:
- Referenced Vision 2030 as Vision 2.0.
- Failed to extract the table; instead, it hallucinated and extracted the text in a different format.
- Failed to extract the number and date of the circular.
Below is the complete extracted text using prompt Extract the complete text including tables and translate it to English
**Subject: Emphasizing the Readiness of Locations and Full Utilization of Absorptive Capacity in Currency Exchange Centers During Ramadan and Hajj Seasons for the Year 1446 AH**
Based on paragraph (b) of Article 14 of the Regulations for the Practice of Currency Exchange, which states that "The Central Bank may issue any instructions it deems necessary to implement supervisory and regulatory requirements for the operations of currency exchange centers," and in line with the main objectives of the Guest Service Program of the Mercy - one of the programs of the Kingdom's Vision 2.0 - regarding the provision of high-quality services to pilgrims and Umrah performers, and achieving a satisfaction rate of 85% for the services provided to pilgrims and Umrah performers by 2025 and 90% by 2030. Given the importance of the currency exchange sector as one of the services provided to pilgrims and Umrah performers in achieving this rate.
Therefore, the Central Bank emphasizes to the currency exchange centers located in Makkah Al-Mukarramah, Madinah Al-Munawwarah, King Abdulaziz International Airport, and the Pilgrims' City in the Governorate of Jeddah, the need to raise the level of readiness and early preparation to provide services for buying and selling foreign currencies and their circulation within the Kingdom in the best possible manner during the seasons of Ramadan and Hajj, to achieve the objectives mentioned above, while adhering to all systems and instructions, for example, and not limited to, the illustrative images in Appendix (1).
For your information and action accordingly.
Yazeed bin Ahmed Al Sheikh
Deputy Governor for Supervision
---
**Appendix (1)**
1. **Preparation and Readiness of Currency Exchange Centers:**
- Ensure the readiness of currency exchange centers in terms of infrastructure, technical systems, and human resources.
- Ensure the availability of sufficient liquidity to meet the needs of pilgrims and Umrah performers.
- Ensure the availability of various foreign currencies in sufficient quantities.
- Ensure the implementation of security and safety measures in currency exchange centers.
- Ensure the availability of trained staff to provide services to pilgrims and Umrah performers.
- Ensure the availability of customer service and complaint handling mechanisms.
- Ensure the implementation of awareness and guidance programs for pilgrims and Umrah performers regarding currency exchange services.
2. **Quality of Services Provided:**
- Ensure the provision of high-quality services to pilgrims and Umrah performers.
- Ensure the implementation of service quality standards in currency exchange centers.
- Ensure the availability of mechanisms for measuring customer satisfaction and service quality.
- Ensure the implementation of continuous improvement programs for services provided.
3. **Compliance with Regulations and Instructions:**
- Ensure compliance with all regulations and instructions issued by the Central Bank regarding currency exchange operations.
- Ensure the implementation of supervisory and regulatory requirements for currency exchange operations.
- Ensure the implementation of anti-money laundering and counter-terrorism financing measures.
- Ensure the implementation of consumer protection measures.
4. **Coordination and Cooperation:**
- Ensure coordination and cooperation with relevant authorities to facilitate the provision of currency exchange services to pilgrims and Umrah performers.
- Ensure coordination and cooperation with banks and financial institutions to facilitate currency exchange operations.
- Ensure coordination and cooperation with service providers in the Hajj and Umrah sector to facilitate the provision of integrated services to pilgrims and Umrah performers.
5. **Monitoring and Follow-up:**
- Ensure continuous monitoring and follow-up of the performance of currency exchange centers during the seasons of Ramadan and Hajj.
- Ensure the implementation of mechanisms for reporting and addressing any issues or challenges faced by currency exchange centers.
- Ensure the implementation of mechanisms for evaluating the performance of currency exchange centers and taking corrective measures if necessary.
Discover more from Shekhar Gulati
Subscribe to get the latest posts sent to your email.
Gemini can be used via Google AI studio.