August 21, 2024

  • Deployed new versions: RSK-M132p4 (v3/text) and RSK-P118p6 (v3/pdf)
  • Fixes post-processing issues related to code recognition and footnote text.
  • Fixes some internal errors.
Single line footnote text was sometimes left unclosed, without a matching closing }. For example, response from the API would return something like:
\footnotetext{
* My single line footnote
This deployment corrects it to:
\footnotetext{
* My single line footnote
}
In some ambiguous cases, we have been omitting the closing triple back ticks which are MMD code delimiters. That has lead to improper rendering in those cases. Most of these situations, if not all, should be fixed by this deployment.

August 20, 2024

  • Deployed new versions: RSK-M132p3 (v3/text) and RSK-P118p5 (v3/pdf)
  • Improved processing of tables with complex content inside cells.
This deployment improves on table OCR when tables have cells with complex content. Here is an example of a very simple table with a complex cell just to illustrate the type of data that these changes affect positively:
First cell of the second row has 3 lines of text and an equation. The table gets processed into MMD like this one:
\begin{tabular}{|c|c|c|}
\hline First column & Second column & Third column \\
\hline \begin{tabular}{l} 
This cell has multiple lines \\
which enables testing tableocr properly. \\
Here goes the equation: \\
\( L=-\frac{1}{N} \sum_{i=0}^{N-1} y_{i} \log \left(\hat{y}_{i}\right) \)
\end{tabular} & Normal cell & Normal cell \\
\hline
\end{tabular}
which renders as:

August 16, 2024

  • Deployed new version: RSK-M132p2 (v3/text)
  • Adds back missing line data formats (data and html)
  • Resolves some cases of incorrect error reporting for v3/latex endpoint

August 15, 2024

  • Deployed new versions: RSK-M132p1 (v3/text) and RSK-P118p4 (v3/pdf)
  • Enables "include_equation_tags": boolean boolean option for v3/text (default is false).
  • Post-processing improvements.

August 14, 2024

  • Deployed new version: RSK-P118p3 (v3/pdf)
  • Fixes diagram post-processing issues.
  • Adds request option include_smiles for digitizing chemistry diagrams (true by default). When false chemistry diagram is preserved as an image.

August 10, 2024

  • Deployed new version: RSK-P118p2 (v3/pdf)
  • Fixes rare post-processing issues related to equations processing.

August 6, 2024

  • Deployed new version: RSK-P118p1 (v3/pdf)
  • Fixes post-processing issues related to tables processing.

July 29, 2024

  • Deployed new versions: RSK-M132 (v3/text) and RSK-P118 (v3/pdf)
  • OCR engine updates

July 26, 2024

  • Deployed new version: RSK-P117p5i4 (v3/pdf)
  • Correctly including equation tags in some edge cases.

July 10, 2024

  • Deployed new versions: RSK-M131p6i3 (v3/text) and RSK-P117p3i2 (v3/pdf)
  • We now move “ 1 t e x t 1 t e x t 1text1 \mathrm { text }” to “ 1 t e x t 1 t e x t 1text1 text” wherever possible. We also have an API option math_fonts_default_to_math to disable this behavior.

June 22, 2024

  • Deployed new version RSK-M131p6i2 (v3/text)
  • Fixes internal errors in v3/text when invalid format is specified in "formats" request argument. Instead of internal error we now return results for all supported formats the were specified in the request, and we ignore the unsupported ones. For details on supported formats see here.

June 19, 2024

  • Deployed new version RSK-M131p4i1 (v3/text)
  • Fixes parsing of chemicals in wrongly rotated images when autorotation is detected correctly.

June 12, 2024

  • Deployed new version RSK-M131p3i1 (v3/text)
  • Fixes wrong items in detection map output, namely contains_chart, and contains_graph

June 9, 2024

  • Deployed new version RSK-P117p1 (v3/pdf)
  • Fixes handling of unrecognized tables in post-processing.

June 7, 2024

  • Deployed new versions RSK-P117 (v3/pdf)
  • This update fixes problems caused by rotated side text detection, improves source code recognition, and recognition of very small lines (e.g. single character line {)

May 31, 2024

  • Deployed new versions RSK-P116p2i1 (v3/pdf)
  • Given $ as inline math delimiter, form fields are recognized without surrounding spaces as $\qquad$ instead of $ \qquad $ because the former has rendering issues

May 30, 2024

  • Deployed new versions: RSK-M131p2i1 (v3/text) and RSK-P116p1i1 (v3/pdf)
  • Post-processing fixes that eliminate several very rare internal errors. We now either return requested content, or no content if there is nothing in the image.

May 27, 2024

  • Deployed new versions: RSK-M131 (v3/text) and RSK-P116 (v3/pdf)
  • Improvements of source code recognition; support for non-English languages in source code
For example, given an image like this:
drawing
returned response now includes correctly recognized Chinese ideographs:
```
def test_function():
    # 测试数字之和
    sum: int = 0
    for i in range(1, 11):
        sum += i
    assert sum == 55
```

May 20, 2024

  • Deployed new version: RSK-M130 (v3/text)
  • Improved subtype chart detection (added analytical subtype)
Requests that specify "include_line_data": true and contain an image such as:
will receive line data as:
[
  {
    "type": "text",
    "cnt": [
      [
        29,
        29
      ],
      [
        29,
        0
      ],
      [
        368,
        0
      ],
      [
        368,
        29
      ]
    ],
    "included": true,
    "is_printed": true,
    "is_handwritten": false,
    "text": "What is the area of the trapezoid?",
    "after_hyphen": false,
    "confidence": 1.0,
    "confidence_rate": 1.0
  },
  {
    "type": "chart",
    "cnt": [
      [
        303,
        331
      ],
      [
        303,
        62
      ],
      [
        681,
        62
      ],
      [
        681,
        331
      ]
    ],
    "included": false,
    "is_printed": true,
    "is_handwritten": false,
    "subtype": "analytical",
    "error_id": "image_not_supported"
  }
]
which has an item with "type": "chart" and "subtype": "analytical".

May 15, 2024

  • Deployed new version: RSK-P115 (v3/pdf)
  • Improved handling of large capital letters at the beginning of paragraphs (see below)
For cases like this one, where the paragraph starts with a very large letter:
we now return correctly recognized text as:
There is now a substantial literature that connects religion and ...rest of text...
with a focus on starting T being handled properly.

May 9, 2024

  • Deployed new versions: RSK-M129p2i1 (v3/text) and RSK-P114p2i1 (v3/pdf)
  • Performance improvements:
    • Faster response times across all endpoints.
    • Large PDFs should now go from loaded to split much faster (splitting time).

May 1, 2024

  • Deployed new versions: RSK-M129p1 (v3/text) and RSK-P114p1 (v3/pdf)
  • Post-processing updates for source code, among others:
    • \rightarrow to ->
    • \Rightarrow to =>

April 30, 2024

  • Deployed new versions: RSK-M129 (v3/text) and RSK-P114 (v3/pdf)
  • Update improves on handwritten Japanese and source code recognition
This update brings initial support for correct source code recognition from images. For example, for an image with source code listing like the following:

Mathpix now returns:
By combining multimethods and packages, we can simplify the abstract syntax tree example by bundling the visitation methods for each tree operation in a package.
```
object TypeChecker {
public:
    void Visit(AssignmentNode & n) {
        // ...
        Visit (n.LHS ());
        Visit (n.RHS ());
        // ...
    }
    void Visit(VariableRefNode & n);
    // ...
};
object CodeGenerator {
public:
    void Visit(AssignmentNode & n);
    void Visit(VariableRefNode & n);
    // ...
};
Node & root = // ...
TypeChecker.Visit(root);
CodeGenerator.Visit(root);
```
which gets renders as:
Similarly, pseudocodes are also wrapped in three back ticks, while Latex with math notation is output. Note, this format for pseudocodes is likely to change in the future.
For pseudocode like the following image:
the output returned looks like:
```
Algorithm 3 An algorithm to divide a given size budget among subexpressions ${ }^{9}$
    func Divide ( $a$ : Arity, $q$ : Size, $l$ : Op. Level, $j$ : Expr. Level, $\alpha$ : Accumulated Args.)
    - Requires: $1 \leq a \leq q \wedge l \leq j$
        if $a=1$ then
            if $l=j \vee \exists\langle x, y\rangle \in \alpha: x=j$ then return $\{(1, q) \diamond \alpha, \ldots,(j, q) \diamond \alpha\}$
            return $\{(j, q) \diamond \alpha\}$
        $L=\{\}$
        for $u \leftarrow 1$ to $j$ do
            for $v \leftarrow 1$ to $(q-a+1)$ do
                $L \leftarrow L \cup \operatorname{DIVIDE}(a-1, q-v, l, j,(u, v) \diamond \alpha)$
        return $L$
```
If one wants to render recognized pseudocode, it is recommended to remove back ticks and replace indents of 4 spaces with $\quad$ . After these transformations the result is:
While Mathpix used to return links of image crop for source code and pseudocodes v3/pdf, that behavior is now changed and v3/pdf now returns MMD outputs as described in this changelog.

April 27, 2024

  • Deployed new versions: RSK-M128p3 (v3/text) and RSK-P113i1p1 (v3/pdf)
    • We now transform \begin{array}{l}{...x...}\\{...y...}\end{array} to \binom{...x...}{...y...}.

April 18, 2024

  • Deployed a new version: RSK-M128p2 (v3/text).
    • Minor post-processing fixes.
  • Deployed new version: RSK-P112 (v3/pdf*)
    • Initial support for detection of in-text placeholders to be filled in (form_field).
    • Algorithm updated with more data.

April 4, 2024

  • Deployed new version RSK-M128 (v3/text)
  • Improved source code and pseudo code indentation
    • source code is enclosed with triple backticks
  • Support for detection of in-text placeholders to be filled in (form_field)
    • placeholder underscores, dots, and dashes are output as \( \\qquad \)
    • placeholder boxes are output as \(\square\)

March 26, 2024

  • Deployed new version: RSK-P111p6i2 (v3/pdf*)
  • Improved export to DOCX: added flattening of nested text tables if the nested table is just one column with several rows.

March 15, 2024

  • Deployed new version: RSK-P111p5i2 (v3/pdf*)
  • MMD to PDF-LaTeX conversion improvements
    • Added additional fonts to support a broader range of scripts.
    • Expanded language detection capabilities to accurately and more reliably handle texts.
    • Language detection extended to include Thai, Tamil, Hebrew, Hindi, and Bengali.
    • Automatically use XeLaTeX for documents containing non-Latin, Greek, Cyrillic (LGC) scripts, ensuring better
      rendering of multilingual texts.
    • Standardized font usage to the Noto family for non-LGC scripts, providing consistency and extensive script support
      across documents.
  • MMD to LaTeX conversions and the tex.zip extension’s output may now include code requiring XeLaTeX for proper rendering of non-LGC texts. It’s notable that only Note to PDF-LaTeX allows for specific font selection. For MMD to LaTeX and the output from the tex.zip extension, conversions standardize on CMU Serif and Noto Serif for non-LGC scripts.

March 13, 2024

  • Deployed a new version: RSK-M127p5i1 (v3/text).
  • Fixed incorrect duplication of the “caret” in Asciimath outputs (e.g., 100(1.03)^2t=5000 instead of 100(1.03)^(^^)2t=5000).
  • Deployed new version: RSK-P111p4i2 (v3/pdf*)
  • Improved export to DOCX: added the ability to automatically detect the language for the entire document.

February 29, 2024

  • Deployed new versions: RSK-M127p4i1 (v3/text and v3/latex) and RSK-P111p3i2 (v3/pdf*)
  • General efficiency improvements.

February 26, 2024

  • Deployed new version: RSK-P111p3i1 (v3/pdf*)
  • Improved export to LaTeX (tex.zip): we now use double backslashes (\\) for line breaks in the Table of Contents and various sections without starting a new paragraph in text mode.

February 22, 2024

  • Deployed a new version: RSK-M127p3 (v3/text).
  • Fixed incorrect grouping of fractional function arguments in Asciimath outputs, fractional function arguments are now properly contained within functions by being wrapped in parentheses (e.g., sec ((5theta)/(4))=2 instead of sec (5theta)/(4)=2 and log ((x)/(y^(5))) instead of log (x)/(y^(5))).

February 16, 2024

  • Deployed a new version: RSK-M127p3 (v3/text and v3/latex).
  • We now validate the region parameter and return appropriate error messages.

February 6, 2024

  • Deployed new version: RSK-P111p2 (v3/pdf*)
  • Added new API option "include_equation_tags": boolean
We’re excited to announce an enhancement to our v3/pdf* endpoints: the ability to include recognized equation numbers directly within the MMD output. Previously, equation numbers were only accessible through the lines.json output format, serving primarily for search purposes without visibility in the main MMD content.
With the new "include_equation_tags": true parameter, our system now integrates equation numbers seamlessly into the MMD output, utilizing the \tag element for clear association. This improvement enriches the MMD files by directly linking equations with their corresponding numbers, facilitating easier reference and navigation.
Here is an example of how the result changes:
For this part of the PDF file
we currently return this MMD:
It is not immediately obvious from Maxwells equations that the time-varying current is the source of radiation. A simple transformation of the Maxwells equations

\[
\begin{aligned}
\nabla \times \mathbf{E} & =-\mu \frac{\partial \mathbf{H}}{\partial t} \\
\nabla \times \mathbf{H} & =\varepsilon \frac{\partial \mathbf{E}}{\partial t}+\mathbf{J}
\end{aligned}
\]

into a single second-order equation either for \(\mathbf{E}\) or for \(\mathbf{H}\) proves this statement. By taking the curl of both sides of the first equation in (1.2) and by making use of the second equation in (1.2), we obtain

\[
\nabla \times \nabla \times \mathbf{E}+\mu \varepsilon \frac{\partial^{2} \mathbf{E}}{\partial t^{2}}=-\mu \frac{\partial \mathbf{J}}{\partial t}
\]

From (1.3), it is obvious that the time derivative of the electric current is the source for the wave-like vector \(\mathbf{E}\).
With the new option enabled ("include_equation_tags": true), we now return:
It is not immediately obvious from Maxwells equations that the time-varying current is the source of radiation. A simple transformation of the Maxwells equations

\[
\begin{align*}
\nabla \times \mathbf{E} & =-\mu \frac{\partial \mathbf{H}}{\partial t}  \tag{1.2a}\\
\nabla \times \mathbf{H} & =\varepsilon \frac{\partial \mathbf{E}}{\partial t}+\mathbf{J} \tag{1.2b}
\end{align*}
\]

into a single second-order equation either for \(\mathbf{E}\) or for \(\mathbf{H}\) proves this statement. By taking the curl of both sides of the first equation in (1.2) and by making use of the second equation in (1.2), we obtain

\[
\begin{equation*}
\nabla \times \nabla \times \mathbf{E}+\mu \varepsilon \frac{\partial^{2} \mathbf{E}}{\partial t^{2}}=-\mu \frac{\partial \mathbf{J}}{\partial t} \tag{1.3}
\end{equation*}
\]

From (1.3), it is obvious that the time derivative of the electric current is the source for the wave-like vector \(\mathbf{E}\).
Besides added equations numbers, there are other differences in the output:
  1. Individual numbered equations are wrapped in \begin{equation*}...\end{equation*} to support numbering.
  2. The aligned environment is now replaced with align* to support numbering
    • We also replace gathered with gather*.
    • In general, we will use the environments that support numbering when using "include_equation_tags": true.
  3. These changes are also visible in lines.mmd.json format.
Here is what the rendering of the recognized portion of PDF page looks like:
Note that there are some limitations in the current implementation:
  1. We do not generate equation references in the text (e.g. \ref{eq:1.3}). In the given example, in the last paragraph, the equation (1.3) is being referenced. While that reference is straight forward, equations (1.2a) and (1.2b) are being referenced as the first equation in (1.2) and the second equation in (1.2). This demonstrates that it takes significant semantic understanding of the document content to correctly unravel all the references, and is beyond the scope of this update.
  2. We do not generate equation labels at the moment. The reasons are:
    • We do not generate \ref elements in text, and that would make \label elements redundant
    • Some widely used LaTeX rendering libraries do not support \label and \ref
  3. When we have a block equation for which we output an array environment, and this block has multiple equation numbers associated with it, only the last equation number is being tagged. This is because LaTeX allows only one equation number per array. The same holds for the cases environment. We do plan to add support for the numcases environment which will enable multiple tags per one block of equations.

February 3, 2024

  • Deployed new ocr-version RSK-M127p2
  • Fixes internal error handling

February 3, 2024

  • Deployed new versions: RSK-M127p1, RSK-P111p1 (v3/pdf*)
  • Improvements to tabular outputs.

February 2, 2024

  • Deployed new versions: RSK-M127, RSK-P111 (v3/pdf*);
  • Initial OCR support Jordan matrices, improvements for tensor indices, and handwritten content;
For example, Mathpix can now correctly recognize matrices as this one:
and return the result:
\( \left(\begin{array}{ccc}\boxed{\begin{array}{cc}18 & -15 \\ 89 & 22\end{array}} & & \\ & \ddots & \\ & & \boxed{\begin{array}{cc}-83 & -6 \\ 9 & 3\end{array}}\end{array}\right) \)

January 27, 2024

  • Deployed new versions: RSK-M126, RSK-P110 (v3/pdf*);
  • OCR algorithm improvements for future chemistry releases;

January 22, 2024

  • Deployed new versions: RSK-M125, RSK-P109 (v3/pdf*);
  • Accuracy improvements of content detection in hard images/pages.

January 17, 2024

  • Deployed new versions: RSK-M124p2;
  • Changed chemistry outputs;

January 16, 2024

  • Deployed new versions: RSK-M124p1;
  • Changed expand_chemistry to include_chemistry;

January 15, 2024

  • Deployed new versions: RSK-M124;
  • OCR improvements for small images;

January 11, 2024

  • Deployed new versions: RSK-M123, RSK-P108 (v3/pdf*);
  • OCR improvements of images with complex tensor indices;
For example, for this image:
the result LaTeX used to be:
\( \epsilon_\sigma^{\mu \nu \rho} \)
For the same image, the new deployment now returns:
\( \epsilon^{\mu \nu \rho}{ }_{\sigma} \)
Here is the difference in rendering previous and current results showing the improvement:

January 9, 2024

  • Deployed new versions: RSK-M122p1;
  • Fixes for internal errors in post-processing when "include_word_data": true is used in the request;

December 28, 2023

  • Deployed new versions: RSK-M122, RSK-P107 (v3/pdf*);
  • Information if a line is printed, handwritten, or both is added to line data outputs;
Text in image or page can be printed, handwritten, or both. We have exposed this information through our line API. For example, for this image:
and request:
{
  "src": "https://mathpix.com/examples/printed_handwritten_0.jpg",
  "formats": [
    "text"
  ],
  "include_line_data": true
}
the response will look like:
{
  "image_width": 808,
  "image_height": 404,
  "is_printed": true,
  "is_handwritten": true,
  "auto_rotate_confidence": 0.0,
  "auto_rotate_degrees": 0,
  "confidence": 1.0,
  "confidence_rate": 1.0,
  "text": "b) Absolute-Value Function:\n\\[\nf(x)=\\frac{1}{2}|x-1|-2\n\\]",
  "languages_detected": [],
  "line_data": [
    {
      "type": "text",
      "cnt": [
        [
          193,
          39
        ],
        [
          380,
          44
        ],
        [
          419,
          50
        ],
        [
          424,
          55
        ],
        [
          430,
          66
        ],
        [
          424,
          105
        ],
        [
          419,
          110
        ],
        [
          259,
          110
        ],
        [
          132,
          105
        ],
        [
          61,
          99
        ],
        [
          39,
          94
        ],
        [
          0,
          77
        ],
        [
          0,
          55
        ],
        [
          55,
          39
        ]
      ],
      "included": true,
      "is_printed": true,
      "is_handwritten": false,
      "text": "b) Absolute-Value Function:",
      "after_hyphen": false,
      "confidence": 1.0,
      "confidence_rate": 1.0
    },
    {
      "type": "math",
      "cnt": [
        [
          44,
          328
        ],
        [
          44,
          157
        ],
        [
          651,
          157
        ],
        [
          651,
          328
        ]
      ],
      "included": true,
      "is_printed": false,
      "is_handwritten": true,
      "text": "\n\\[\nf(x)=\\frac{1}{2}|x-1|-2\n\\]",
      "after_hyphen": false,
      "confidence": 1.0,
      "confidence_rate": 1.0
    }
  ]
}
In the line data, sentence has is_printed true and is_handwritten false, while the math has is_handwritten true and is_printed false.
Sometimes, the content can be both printed and handwritten at the same time. Here is one such example:
For the request:
{
  "src": "https://mathpix.com/examples/printed_handwritten_1.jpg",
  "formats": [
    "text"
  ],
  "include_line_data": true
}
the response will look like:
{
  "image_width": 354,
  "image_height": 120,
  "is_printed": true,
  "is_handwritten": true,
  "auto_rotate_confidence": 0.0,
  "auto_rotate_degrees": 0,
  "confidence": 0.99951171875,
  "confidence_rate": 0.99951171875,
  "latex_styled": "\\frac{9}{12}+\\frac{2}{11}=\\frac{99}{132}+\\frac{24}{132}=\\frac{123}{132}",
  "text": "\\( \\frac{9}{12}+\\frac{2}{11}=\\frac{99}{132}+\\frac{24}{132}=\\frac{123}{132} \\)",
  "languages_detected": [],
  "line_data": [
    {
      "type": "math",
      "cnt": [
        [
          16,
          110
        ],
        [
          16,
          1
        ],
        [
          338,
          1
        ],
        [
          338,
          110
        ]
      ],
      "included": true,
      "is_printed": true,
      "is_handwritten": true,
      "text": "\\( \\frac{9}{12}+\\frac{2}{11}=\\frac{99}{132}+\\frac{24}{132}=\\frac{123}{132} \\)",
      "after_hyphen": false,
      "confidence": 0.99951171875,
      "confidence_rate": 0.9999904235654993
    }
  ]
}
It can be seen in the line_data that the line object has both is_printed and is_handwritten set to true.
In v3/pdf* endpoint(s) lines.json and lines.mmd.json outputs are updated with new is_printedand is_handwritten fields for each line.

December 13, 2023

  • Deployed new versions: RSK-M121, RSK-P106 (v3/pdf*);
  • Added support for the 12 new LaTeX commands;
  • Improved recognition of text lines with many dots (e.g. text lines in table of contents images);
  • Improved recognition of languages that use Cyrillic alphabets.
We added recognition support for 12 new LaTeX commands that were previously unsupported.
\xrightarrow
\( \xrightarrow{\mathrm{H}^{\oplus} / \mathrm{H}_{2} \mathrm{O}} \)
\xlongequal
\( E_{2} \xlongequal[r_{4}+r_{8}]{\substack{r_{9}-r_{5} \\ c_{8}-c_{5}, \ldots, c_{5}+c_{5}}}\left|\begin{array}{cccc}-1 & 7 & -4 & -1 \\ -2 & 0 & -4 & -2 \\ -4 & -4 & 8 & -1 \\ -9 & -2 & 6 & 2\end{array}\right| \)
\gtrdot
\( \frac{1}{g_{4}^{2}} \lessdot-\frac{2 k N_{c} \ln (v)}{g_{5}^{2}} \gtrdot 0 \)
\boxtimes
\( (u \boxtimes v) \cdot(w \boxtimes x)=(v \cdot w)(u \boxtimes x) \)
\downharpoonleft
\( u_{r r} \downharpoonleft \frac{\sigma_{r r}}{E} \)
\downharpoonright
\( \int \rho_{i k} d f_{i} \downharpoonright 0 \)
\leftharpoondown
\( (u \nabla) v \leftharpoondown-\frac{1}{\rho} \nabla p+\nu \Delta v \)
\leftharpoonup
\( \phi(u, v) \leftharpoonup \arctan \left[\frac{I(u, v)}{R(u, v)}\right] \)
\rightharpoondown
\( \zeta \rightharpoondown i\left(\frac{\pi}{2}+\theta\right) \)
\rightharpoonup
\( f(t) h(t) \rightharpoonup H(\mu) \div F(\mu) \)
\upharpoonleft
\( j \upharpoonleft-\frac{2 \mu \alpha_{1} I_{1}}{v_{0}} \)
\upharpoonright
\( W \upharpoonright N f\left(\frac{S}{N}, P\right) \)

December 7, 2023

  • Deployed new version (v3/pdf), RSK-P105;
  • Another improvement for table of contents pages.

November 28, 2023

  • Deployed new version (v3/pdf), RSK-P104;
  • Improved processing of table of contents pages.

October 31, 2023

  • Deployed new version (RSK-M120);
  • Chart detection;
This is a first step towards recognition of charts. Basic charts detection is supported in images.
When "include_line_data": true is specified in the request, for all the charts we now return correct chart type in line data response instead of generic diagram. Currently, several types of charts are being detected:
  1. column chart
  2. bar chart
  3. line chart
  4. pie chart
  5. area chart
  6. scatter chart
The type of chart is available as "subtype": ... field of the line object. This list will be extended with additional categories in the future.
  • Algorithm pseudo code recognition improvements.
We used to fail recognition for images with complex algorithm pseudocode like this one:
For such image, the returned MMD now looks like this:
Algorithm 1 Gradient Sign Dropout Layer (GradDrop Layer)
choose monotonic activation function \( f \quad \triangleright \) Usually just \( f(p)=p \)
choose input layer of activations \( A \quad \triangleright \) Usually the last shared layer
choose leak parameters \( \left\{\ell_{1}, \ldots, \ell_{n}\right\} \in[0,1] \quad \triangleright \) For pure GradDrop set all to 0
choose final loss functions \( L_{1}, \ldots, L_{n} \)
function \( \operatorname{BACKWARD}\left(A, L_{1}, \ldots, L_{n}\right) \quad \triangleright \) returns total gradient after GradDrop layer
for \( i \) in \( \{1, \ldots, n\} \) do
calculate \( G_{i}=\operatorname{sgn}(A) \circ \nabla_{A} L_{i} \quad \triangleright \operatorname{sgn}(\mathrm{A}) \) inspired by Equation 3
if \( G_{i} \) is batch separated then
\( G_{i} \leftarrow \sum_{\text {batchdim }} G_{i} \)
calculate \( \mathcal{P}=\frac{1}{2}\left(1+\frac{\sum_{i} G_{i}}{\sum_{i}\left|G_{i}\right|}\right) \quad \triangleright \mathcal{P} \) has the same shape as \( G_{1} \)
sample \( U \), a tensor with the same shape as \( \mathcal{P} \) and \( U[i, j, \ldots] \sim \operatorname{Uniform}(0,1) \)
for \( i \) in \( \{1, \ldots, n\} \) do
calculate \( \mathcal{M}_{i}=\mathcal{I}[f(\mathcal{P})>U] \circ \mathcal{I}\left[G_{i}>0\right]+\mathcal{I}[f(\mathcal{P})<U] \circ \mathcal{I}\left[G_{i}<0\right] \)
set newgrad \( =\sum_{i}\left(\ell_{i}+\left(1-\ell_{i}\right) * \mathcal{M}_{i}\right) \circ \nabla_{A} L_{i} \)
return newgrad
and it renders as:
While the correct indentation is still missing, the content is now being returned instead of Content not found error.

October 30, 2023

  • Deployed new version, RSK-P103p6 (v3/pdf*);
  • Added support for remove_section_numbering and preserve_section_numbering. Default behavior is changed
    to preserve_section_numbering: True.
*Note that only one of auto_number_sections, remove_section_numbering, or preserve_section_numbering can be true at a time.

October 27, 2023

  • Deployed new versions: RSK-M119p7, RSK-P103p5 (v3/pdf*);
  • Post-processing improvements: we now return result instead of Internal error for some classes of challenging inputs.

October 21, 2023

  • Deployed new version, RSK-P103p3 (v3/pdf*);
  • Fixed new line inconsistencies that occurred when content from different pages is joined;
  • Single new line is generated before \footnotetext instead of two new lines;
  • Two new lines introduce a new paragraph, and that behavior was breaking rendering experience in some cases.

October 19, 2023

  • Deployed new version, RSK-M119p6;
  • False paragraph detection fixed for images with list items that start with Cyrillic letters.

October 18, 2023

  • Deployed new version, RSK-M119p5;
  • This update is focused towards more accurate separation of text into paragraphs.

October 14, 2023

  • Deployed new versions: RSK-M119p4, RSK-P103p2 (v3/pdf*);
  • Fixed whitespace between Chinese symbols which used to be generated when the adjacent text lines are joined.
As an example, for image like this one:
the result used to look like:
【甲】孔子说: “我十五岁开始有志于做学问,三十岁能独立做事情,四十岁能 (通达事理) 不被外物所迷惑,五十岁能知道哪 *extra whitespace* 些是不能为人力所支配的事情,六十岁能听得进不同的意见,到七十岁才做事能随心所欲,不会超过规矩。”
Result obtained with this version does not contain extra whitespace and look like this:
【甲】孔子说: “我十五岁开始有志于做学问,三十岁能独立做事情,四十岁能 (通达事理) 不被外物所迷惑,五十岁能知道哪些是不能为人力所支配的事情,六十岁能听得进不同的意见,到七十岁才做事能随心所欲,不会超过规矩。

October 13, 2023

  • Deployed new version (v3/pdf*), RSK-P103p1;
  • Improved ordering of answers to multiple choice questions in some cases. For this part of PDF page:
we used to return the answers in height determined order:
10. Demostrar que si \(a, b \in \mathbb{R}\), entonces:
a) \(|-a|=|a|\).
e) \(|a-b| \leq|a|+|b|\)
b) \(\sqrt{a^{2}}=|a|\).
f) \(||a|-| b|| \leq|a-b|\).
c) \(|a-b|=|b-a|\).
g) \(a \neq 0,\left|\frac{1}{a}\right|=\frac{1}{|a|}\).
d) \(\left|a^{2}\right|=|a|^{2}\).
h) \(b \neq 0,\left|\frac{a}{b}\right|=\frac{|a|}{|b|}\).
Now, we return the answers in the correct order:
10. Demostrar que si \(a, b \in \mathbb{R}\), entonces:
a) \(|-a|=|a|\).
b) \(\sqrt{a^{2}}=|a|\).
c) \(|a-b|=|b-a|\).
d) \(\left|a^{2}\right|=|a|^{2}\).
e) \(|a-b| \leq|a|+|b|\)
f) \(||a|-| b|| \leq|a-b|\).
g) \(a \neq 0,\left|\frac{1}{a}\right|=\frac{1}{|a|}\).
h) \(b \neq 0,\left|\frac{a}{b}\right|=\frac{|a|}{|b|}\).

October 12, 2023

  • Deployed new version, RSK-M119p3;
  • Accuracy improvements of Korean and handwritten Chinese;
  • Fixes several issues with braces recognition.
Here is an example image:
Result before (missing outer braces):
Result with current version:

October 9, 2023

  • Deployed new version (v3/pdf*), RSK-P103;
  • more reliable recognition of footnote text;
  • initial support for table of contents;
  • pseudo code algorithms are now searchable;
    • individual lines from pseudo code algorithms will be a part of lines.json output.

September 5, 2023

  • Deployed new version (v3/pdf*), RSK-P101;
  • Added basic support for text in the footnote section of the page. Instead of breaking the main flow, especially in multi-column documents, the text will be wrapped inside \footnotetext{ ... }.

May 25, 2023

  • Deployed new version, RSK-115;
  • Improved quality of printed and handwritten Chinese recognition.

April 12, 2023

  • Deployed new version, RSK-113;
  • This update is focused on chemistry recognition.
As a reminder, when "include_smiles": true is a part of the request, Mathpix can recognize chemistry diagrams such as:
and return the SMILES representation which looks like this:
<smiles>Cn1c(=O)c2c(ncn2C)n(C)c1=O</smiles>
The list of improvements:
  • support for stereochemistry:
    • for example, the image
is transcribed to: <smiles>O=S(=O)(c1ccc(F)cc1)N1C[C@@H](O)[C@H](N2CCCC2)C1</smiles>;
  • support for Markush structures:
    • for example, the image
is transcribed to <smiles>[Z2]Nc1c(CC([R10])CSC)ncn1CC#C</smiles>;
  • basic support for superatoms:
    • more superatoms will be supported in future;
  • significant recognition accuracy improvement for both handwritten and printed chemistry diagrams.

February 22, 2023

  • Deployed new version, RSK-111;
  • Added support for the new table recognition algorithm;
  • Minor general changes needed to properly support the algorithm.
A new table recognition algorithm is available in the v3/text and v3/pdf endpoints. It can be enabled by specifying "enable_tables_fallback": true as one of the request arguments.
We care deeply about backwards compatibility. The new algorithm will only be used if both of the following conditions are fulfilled:
  • Our standard algorithm failed to recognize a table;
  • "enable_tables_fallback": true is specified as a request argument.
We have ensured that there will be no computational overhead for customers who do not specify this option, so response times will not be affected.
We have invested in a hybrid approach that will be able to tackle complicated cases like:
  • extremely large tables (e.g. tables with hundreds of cells);
  • tables with very complex structures (e.g. tables with many \multirow and \multicolumn cells);
  • tables featuring table cells with complex content like:
    • table cells with complex math like large matrices or several aligned equations;
    • tables cells containing whole paragraphs of text;
  • tables containing text in languages that are more challenging to recognize properly compared to English:
    • this includes languages with rich alphabets like Chinese, Japanese, Hindi, Hebrew, Arabic, and others;
  • tables containing cells with rotated text;
  • tables containing diagrams inside cells like:
    • table cells with chemistry diagrams (note that these can be converted to SMILES);
    • table cells with natural images or similar:
      • table will still be recognized and contain the image link for the diagram inside its cells.
We will also support all combinations of the above cases.
The algorithm we are releasing now might still struggle with:
  • tables with cells containing complex math like large matrices or several aligned equations;
  • tables containing diagrams inside cells;
  • empty grids of cells without textual content;
  • tables with rotated text are partially supported:
    • in v3/pdf cells containing rotated text will be embedded as images.
We will add improvements that will cover specified cases shortly.
Some differences in output produced by the new algorithm compared to the standard one:
  • Column alignment is always central;
  • All cells have all borders (top, bottom, right, and left).

February 1, 2023

  • Deployed new version, RSK-110;
  • Added support for new LaTeX commands: \measuredangle, \grave, \bumpeq, and \amalg;
  • Improved recognition of constructs expressed with \lceil, \rceil, \lfloor, and \rfloor in combination with \left and \right;
  • Improved formatting of equations in text mode (see bellow for details);
  • Improved recognition of equations that contain large sub-equations in subscripts;
  • Improved recognition of handwritten French;
  • Improved recognition of handwritten German;
  • Improved recognition of handwritten Chinese;
  • Improved recognition of handwritten Japanese;
  • General improvements (new data iteration).
The default “text” output has been changed, for example see the following equation:
from the current “text” output:
\( y=mx+b \)\n\( x=y^{2}-1 \)
and two asciimath equation outputs, to:
\( \begin{array}{l} y = mx + b \\ x=y^{2}-1 \end{array} \)
with one single asciimath equation output:
{:[y=mx+b],[x=y^(2)-1]:}
which will make the “text” derived formats more consistent with what is currently returned for equations with a left brace:
which currently yields this “text” output:
\( \left\{\begin{array}{l}2 x+8 y=21 \\ 6 x-4 y=14\end{array}\right. \)
and this “asciimath” output:
{[2x+8y=21],[6x-4y=14]:}
Since we are already emitting in certain cases (eg, equations with no braces aligned around the “=” sign instead of being left aligned) asciimath for v3/text that looks like:
{:[2x+8y=21],[6x-4y=14]:}
we consider this update to be an inconsistency bugfix instead of a new feature with the potential to break backwards compatibility.
In general, it is desirable for our API for small changes in input to result in small changes in output. For example, removing the left brace from equation 2 will simply change the v3/text asciimath from:
{[2x+8y=21],[6x-4y=14]:}
to:
{:[2x+8y=21],[6x-4y=14]:}
which is a smaller change than the previous behavior, in which subtracting a left brace results in two equations instead of 1.

January 30, 2023

  • Deployed new version, RSK-109;
  • Improved recognition of isolated symbols.

January 12, 2023

  • Deployed new version, RSK-108;
  • Improved handling of images that contain mixed math and text in Russian.

December 2, 2022

  • Deployed new version, RSK-107p2;
  • Changes in spacing of arrays, aligned arrays and similar, & and \\ now always have spaces around them (even with rm_spaces in the request);
  • Visually unpleasant blocks of equations are being converted to left alignment instead of keeping the wrong alignment.

November 25, 2022

  • Deployed new version, RSK-107;
  • Improvements related to worksheet crops, small images with strong or dashed border near the content.

November 22, 2022

  • Deployed new version, RSK-106;
  • Improvements related to formatting of references in PDF pages, especially pages with green/red link boxes.

November 15, 2022

  • Deployed new version, RSK-105;
  • General improvements to handling zoomed out and zoomed in images. No changes to output formatting or error characteristics.

November 14, 2022

  • Deployed new version RSK-104p1;
  • Formating of block math is fixed in certain cases where the equations were wrongly kept in the text mode.

November 11, 2022

  • Deployed new version RSK-104;
  • Incremental improvement of image parsing module. Includes fixes for images with many lines of text. Accuracy improvements on handwritten data. No changes to output formatting.

November 3, 2022

  • Deployed new version RSK-103p1;
  • Fixed string post processing issues.
In this version, we have changed the default Markdown / LaTeX for the following character:
# -> \# 
While # works fine in Markdown and has the same behavior as \#, the former causes LaTeX compilation issues, whereas \# succeeds in LaTeX without any problem. We chose to always emit LaTeX \# instead of # so that our output would be more compatible and less likely to cause issues. The updated character \# is compatible with Markdown as well as LaTeX.
Unescaped # will simply no longer appear in OCR Markdown / LaTeX outputs.
Alternative math formats such as Asciimath are not affected by the change, this is a Markdown / LaTeX change only.

October 21, 2022

  • New enable_spell_check option to the v3/text and v3/pdfs endpoints greatly improves handwriting OCR for English (other languages coming soon).

June 6, 2022

  • Resolved a critical bug that impacted PDF processing of two-column PDFs;
  • You can now request that only certain subsets of pages are processed in a PDF, via the new page_ranges field;
  • Pushed latency improvements that benefit all endpoints, reducing processing time by 30% on average;
  • You can now query hour by hour API usage using the following endpoint.

April 27, 2022

  • Updated how line data is represented for PDFs from using rectangular regions to polygonal contours (this is helpful for handwritten PDFs where text lines are generally not rectangular);
  • Added page dimensions to the line-by-line data structures;
  • There are two available data structures for line-by-line data:
    • Raw PDF lines data: this is the ideal data structure for searching; does not contain contextual annotations for titles, abstracts, etc;
    • Context enhanced PDF mmd lines data: you can use this to re-create the full document, including contextual annotations for titles, abstracts, etc. (see here for syntax);
  • Published a Github repo which contains client-side code for live
    drawing with the Mathpix digital ink API containing a fully working example of leveraging user actions like scribbling
    and strikethrough to delete content.

April 18, 2022

  • Added an EU server region (AWS region eu-central-1) to decrease latencies for European customer and also for adherance to GDPR:
  • You can now use app-tokens for authenticating requests inside client side app code;
  • The new app-tokens route provides a include_strokes_session_id flag, which when true, returns a strokes_session_id string that can be used inside calls to v3/strokes, enabling digital ink sessions with live updates:
    • Pricing for the strokes endpoint when using session_ids can be found here;
  • Add OCR support for basic handwritten PDFs.

March 28, 2022

  • You can now get detailed line-by-line data for PDFs, including geometric coordinates, via the new GET v3/pdf/<pdf_id>.lines.json endpoint;
  • Better robustness for our v3/text endpoint:
    • Our ability to correctly interpret complex layouts involving math and text has improved, with much-improved edge case handling and handling of line text for skewed images and other image distortions that occur frequently in consumer photo search applications.

March 14, 2022

  • Our new OCR models feature stringent guarantees of syntactic correctness, resolving a rare but long-standing problem of occasionally malformed LaTeX strings, resulting in rendering errors due to double subscripts, double superscripts, malformed tables, and other syntax issues. This has been fixed at a fundamental level. Syntax issues are essentially completely fixed;
  • Deprecated \atop command in favor of \substack.

February 8, 2022

We have recently switched to a new, faster database to save image and PDF data. Next week, we will decommission our old database. This will result in OCR API image results log data from before December 1st, 2021 becoming unavailable via the GET v3/ocr-results endpoint. Note that we have already migrated all PDF data to the new database, so there will be no data loss for PDF data.

November 15, 2021

  • Deployed incremental update to our re OCR engine, resulting in:
    • significantly improved handwriting recognition, including disambiguating symbols based on context;
    • improved table parsing accuracy;
    • notably fewer errors.

September 2, 2021

  • Deployed a core algorithm update for our image parsing module, resulting in significantly better accuracy and edge case behavior for all endpoints;
  • Updated CLI with additional conversion types;
  • Significant improvements to chemical diagram OCR;
  • Support for asynchronous image processing.

July 27, 2021

  • Added support for Tamil, Telugu, Gujarati, and Bengali;
  • Updated our OCR to use a more effective representation of Chinese characters, leading to higher accuracy, and better coverage;
  • Added support for \bigcirc.

July 19, 2021

  • Support for sending image binaries for lower image upload latencies;
  • Support for tags which allow you to associate an attribute with your requests and subsequently retrieve the associated requests by using tags in a /v3/ocr-results query;
  • PDF processing updates:
    • Fixed a bug where pages were getting skipped;
    • Improved processing of PDFs with foreign languages;
    • Added support for configurable math delimiters.

April 12, 2021

  • Servers in Singapore for faster latencies for API customers in Asia;
  • Triangle diagram OCR now supported for diagrams commonly found in trigonometry textbooks;
  • Added InChI option for chemical diagram OCR.

April 2, 2021

  • Added a include_word_data parameter to the v3/text endpoint, which when set to true, returns word by word information, with separate results, confidences values, and contour coordinates for each word;
  • Beta printed chemical diagram OCR to return SMILES format.

March 10, 2021

  • New v3/pdf API endpoint (beta);
  • PDF conversion CLI tool;
  • Fixed miscellaneous bugs in v3/text processing for messy images;
  • Incremental improvements to handwriting recognition and printed table recognition for all endpoints;
  • Added support for the following printed characters:
\nvdash
\nvDash
\bigvee
\bigwedge
\fallingdotseq
\risingdotseq
\mho
\unlhd
\unrhd
\overleftrightarrow
\bigodot

February 7, 2021

January 5, 2021

  • Improved math handwriting recognition;
  • Improved printed Romanian, Polish, Serbian, Ukrainian recogntion;
  • Added support for the following LaTeX characters:
\varsigma
\llcorner
\lrcorner
\uplus
\biguplus
\triangleright
\triangleleft
\dddot
\circledast
\rtimes
\ltimes
\supseteqq
\oiiint
\supsetneqq
\rightsquigarrow
\Vdash
\models

December 1, 2020

November 12, 2020

  • Added autorotation for v3/text.
Images like this now work in v3/text:
The goal of automatic rotation is to pick correct orientation for received images before any processing is done. The result of auto rotation looks like:
We will soon add these features to v3/latex and v3/batch as well. We implemented a very conservative rotation confidence threshold, meaning you should still try to call the API with a properly oriented image if possible!
API docs on autorotation:

November 9, 2020

  • v3/text general improvements;
First of all, we trained our models on a larger dataset, resulting in a general accuracy increase.
Secondly, we improved the precision of predictions, at the potential cost of slightly decreasing prediction recall in some circumstances. Here is an example of an image, where previously our v3/text tried to read the bottom, cut off parts of the image:
Now, v3/text ignores these sections, resulting in a much cleaner output than before. The endpoint will still try to read everything in an image (vs the v3/latex endpoint which tries to read the main equation), but will be slightly less aggressive in reading unusual image sections in order to avoid garbage outputs.
  • Chemistry diagram detection;
We have added a new field in our LineData object, subtype, so that we can return more information about diagrams to API clients. Currently subtype can only be chemistry, but more diagram subtypes are coming soon.
See an example request with chemistry: https://mathpix.com/docs/ocr/examples#chemistry-diagrams.
  • Added ability to create and disable new API keys to accounts.mathpix.com OCR dashboard;
  • Added ability to invite users (to have access to API keys, usage statistics, image results dashboard) to
    accounts.mathpix.com OCR dashboard.

October 14, 2020

  • Added support for the following characters:
«
»
\gtrless
\lessgtr
\nsucceq
\npreceq
\llbracket
\rrbracket
\curvearrowright
\curvearrowleft
№
  • Improved accuracy on:
    • Handwriting (math and text);
    • Hindi language recognition (printed);
    • Tables and matrices;
  • We now support backgroundless PNGs with alpha channels.

August 14, 2020

August 1, 2020

July 13, 2020

  • Replace \dots with either \ldot or \cdot;
  • Predict empty braces when appropriate, like in Chemistry images (e.g. {}^);
  • Fix v3/text bug where very wide lines of text were getting skipped;
  • Improved accuracy on handwritten math;
  • Improved accuracy on table predictions;
  • Improved accuracy on photo images of printed Hindi and Chinese text;
  • Add support for \mid predictions inside set notation;
  • Add support for the following languages: Czech, Turkish, Danish.

July 9, 2020

July 1, 2020

  • Skip diagrams in v3/text which caused garbage results.
Yet another reason to use v3/text over v3/latex! v3/text intelligently skips diagrams!