README.md

# Development documentation

## The manager logic

The manager is responsible for detecting the correct root file and once detected to parse the whole project. Its logic is shown as follows.
```mermaid
graph TD
  A[OnWatchedFileChanged] --> B[ParseFileAndSubs];
  C([After finding a new root file]) -->|On the root file| B;
  C --> D[ParseFls];
  E([After a successful build]) --> D;
  D -->|For new files| B;
  B -->|For every file| F[ParseInputFiles];
  B -->|For new files| G[addToFileWatcher];
  F -->|For new files| B
```

## Scripts for Intellisense

These scripts are actually only frontend to the `pyintel` package, which implements the core mechanisms. It fetches the latest list of packages and classes along with their descriptions from CTAN at https://ctan.org/pkg. It uses the TeXLive Package database `texlive.tlpdb` retrieved from https://mirrors.ircam.fr/pub/CTAN/systems/texlive/tlnet/tlpkg/texlive.tlpdb.

### ctanpkglist.py

It produces the intellisense data for classes and packages and save the result as json files: `../data/classnames.json` and `../data/packagenames.json`.

#### Classes

The list of classes is computed from the TeXLive Package database by looking for all `.cls` files. The description associated to each class is obtained from CTAN.

#### Packages

Getting a proper list of packages is tricky as the package names (as listed by CTAN) do not always match the base names of the `.sty` files to be loaded by `\usepackage`. This is handled in the following way

- We use the TeXLive Package database to
  - List all the `.sty` files provided by TeXLive
  - For every package, list the `.sty` files it contains. The name to pass to `\usepackage` is the base name of one of the `.sty` files listed there.
- For every package `pkg` listed by CTAN
  - If `pkg.sty` exists in the TeXLive Package database, store `pkg` for package intellisense.
  - If not, search if a package `pkg/` exists in the TeXLive Package database and look up a file whose lowercase name matches `pkg`. If it is found, then save it for package intellisense.

As some packages cannot be properly detected using the above mechanism, we maintain a list of extra packages to be added to the list in [extra-packagenames.json](extra-packagenames.json). These packages are automatically added at the end of the [`ctanpkglist.py`](ctanpkglist.py) script.

### unimathsymbols.py

It parses uni-math symbols from http://milde.users.sourceforge.net/LUCR/Math/data/unimathsymbols.txt and save the result as a json file. The result is used to generate command intellisense.

### parse-cwl.ts
```
ts-node parse-cwl.ts both|all|ess|essential|<package>.cwl
```

This script generates intellisense data from the files listed in `cwl.list` file (`both`, `ess`, `essential`) or all files (`both`, `all`, `<package>.cwl`) in `cwl/` folder, depends on the command line argument provided to the script. The generated intellisense `.json` file is placed at `../data/packages` (`both`, `ess`, `essential`) or `packages` (`both`, `all`, `<package>.cwl`).
Further, `expl3.cwl` file generated by `latex3command.py` is also parsed to `expl3.json` and placed at `../data/packages`.
Note that the directory must already exist. It is recommended to first run the script `getcwl.sh` to download live raw data from https://github.com/texstudio-org/texstudio. Note that this requires subversion installed.

If you have a custom `<package>.cwl` file, you can put it in `LaTeX-Workshop/dev/cwl` folder. Then run `ts-node parse-cwl.ts <package>.cwl` in command line. Finally, the generated `<package>.json` file can be found in `LaTeX-Workshop/dev/packages`. You may need to manually create this folder first to run the script.

For every package or class, one `.json` file is generated, containing the data for the package defined in the `.cwl` file. This file has the following structure
```typescript
{
  deps: {
    name: string // The name of other package to also include along with this one
    if?: string // But only when package is used with this option
  }[]
  macros: MacroRaw[] // All macros of this package
  envs: EnvironmentRaw[] // All environments of this package
  keys: { [key: string]: string[] } // The key-value pairs that the macros/environments or this package can use
  args: string[] // The key of key-value pairs defined above, where the pairs can be used as package options
}
```

In this `json` object, the macros have the following structure:
```typescript
{
  name: string // The name of the macro, without the argument signature e.g., {}
  arg?: {
    format: string // The argument signature
    snippet: string // The snippet to insert besides macro name. This variable does not include name
    keys?: string[] // The key of key-value pairs defined in the package, where the pairs can be used as possible macro argument values
    keyPos?: number // The index of argument which may take the key-value pair. Start from zero, count from left to right, regardless of argument types e.g., {}, [], () or ||.
  }
  if?: string // The macro is available only when package is used with this option
  unusual?: boolean // The macro is an unusual one
  detail?: string
  doc?: string
}
```
An example is:
```json
{
  "name": "DeclareAcronym",
  "arg": {
    "format": "{}{}",
    "snippet": "DeclareAcronym{${1:id}}{${2:properties%keyvals}}",
    "keys": [
      "\\DeclareAcronym#c,..."
    ],
    "keyPos": 1
  }
}
```

The environments have the following structure:
```typescript
{
  name: string // The name of the env, without \begin{}
  arg?: {
    format: string
    snippet: string
    keys?: string[]
    keyPos?: number
  }
  if?: string
  unusual?: boolean
}
```
An example is:
```json
{
  "name": "acronym",
  "arg": {
    "format": "[]",
    "snippet": "acronym[${1:longest}]"
  }
}
```

Completion files for classes are all prefixed by `class-`.

### latex3command.py

This script generates `expl3.cwl` for LaTeX-3 from all `.dtx` files defined in `/opt/texlive/2024/texmf-dist/source/latex/l3kernel/`, which can be overwritten by setting environment variable `PATH_TO_DTX`.

## Grammar files

- `update-grammar.js` retrieves the latest grammar files from https://github.com/jlelong/vscode-latex-basics.
-`build-grammar.js` generates the `.json` grammar files from `.yaml`.