IE 11 not showing the characters properly when font download is disabled in PDFJS

Due to font download being disabled the characters generated inside PDFJS are displayed totally wrong. while trying to fix the issue, I tried multiple things, like upgrading my PDFJS code and also tried to include a font library inside the PDFJS folder to check if it solves the issue but it didn't either. So I am all out of ideas and any help would be great.

I have provided the machine details in which I ran PDFJS, along with its version and also have attached the sample file and the expected and actual results below.

Link to PDF file: sample.pdf

Configuration:

Web browser and its version: IE 11
Operating system and its version: Windows 7 Virtual Machine
PDF.js version: 1.4.20
Is an extension: No

Steps to reproduce the problem:

1. Open IE 11
2. Open settings and then go to Internet Options
3. Go to security tab and click on customize level... button
4. Scroll down to the section Downloads
5. Under Downloads disable Font download
6. Click on Ok and then click on Apply.
7. Open the above attached PDF, you would see that the horse that is getting displayed at the bottom of the page in other browsers would be replaced with 123 in font download disabled IE 11.

What is the expected behavior?

What went wrong? (add screenshot)

Have used viewer.html to render the attached pdf file.

I have raised the same issue in PDFJS github repo (https://github.com/mozilla/pdf.js/issues/7424).

Regards Sumit