This ongoing Docker Labs GenAI series will explore the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing things as open source so you can play, explore, and hack with us, too.
In a previous article in this series, we discussed linters, and we talked about running ESLint with an AI assistant. In this post, we will cover AI-assisted violation resolutions.
Before an AI assistant can take action on violations via the editor or Git (Figure 1), we first need to answer the question: Can an LLM generate resolutions for ESLint violations with the right knowledge and tools?
Fix any lint violations you can in my project.
ESLint violations
To resolve a violation, the LLM needs information about the violation from the linter, and some context about the source code causing it. A level of human supervision may also be necessary. We want to optimize to shrink context, because models both limit context windows and yield better results with less context. This makes us wonder: “What is the minimum amount of context and supervision necessary to resolve a lint violation?”
For example, without human instruction, our assistant can’t know how to balance our parentheses:
// export const dockerfileRegex = new RegExp('(.*\\.?Dockerfile', 'i');
// Goal: new RegExp('(.*)\\.?Dockerfile', 'i');
Your `dockerfileRegex` regular expression is currently unterminated. This means the group you started at `(.*\\.?Dockerfile` doesn't have an ending bracket. Here is a fixed version of it:
```typescript
export const dockerfileRegex = new RegExp('(.*\\.?Dockerfile)', 'i');
```
I added a ")" to close the group.
Focusing on certain violations
We want to narrow the list of ESLint rules to violations the LLM can help with.
In Figure 2, we have compiled every ESLint rule that:
- Is recommended
- Is not auto-fixable by ESLint itself
And we want rules that:
- Do not require supervision to make reasonable fixes.
This context yields four total groups of violations:
Group 1 (No context needed from the surrounding source code or supervision needed.) Scripts can be applied to fix these violations.
Group 2 (No context needed from the surrounding source code, but these might be mistakes that necessitate the LLM to evaluate some of the code.) LLM can generate scripts to resolve these violations, but we would be assuming violations are mistakes. There are some violations that the LLM can evaluate on its own.
Group 3 (Context is needed from surrounding source code, but no supervision is necessary.) This is the best opportunity for our LLM to provide help. We can use tools to pull the right context in.
Group 4 (Context is needed from surrounding source code and supervision might be needed.)
- Invalid Regex
- Unsafe optional chaining
- Constant condition
Depends a lot on the exact situation, but LLMs may be useful?
Thankfully, nearly all the violations could have reasonable fixes applied without supervision. These are the violations that we will focus on.
Initial prompts
First, we create the prompts to attempt to fix ESLint violations.
You are an AI assistant who specializes in resolving lint violations in projects. Use the tools available to quickly take action and be very brief.
1. Run linter.
2. Evaluate total violations.
// What to do?
Unfortunately, we run into a roadblock when it comes to learning about lint violations from our ESLint tool. When using summary output, we don’t have enough information to know what we’re fixing. However, when using JSON output, we found that as few as 100 violations in a project caused ESLint to send over 10,000 characters over standard out. That would be a problem since many models at current limit us to 4-8k tokens. We need a way to persist this large output but without consuming tokens.
Artifacts
While we want to use ESLint, it can easily balloon an AI conversation past the model’s context size. To address this, we modified the ESLint tool to write eslint.json to a new ephemeral Docker volume. Reading and writing to this volume serves as another conversation between tools, but sandboxed away from the LLM’s context-sensitive function call output.
We added tools to consume the ESLint artifact in different ways depending on the number of violations summarized.
- name: parse_lint_results
description: Loads lint violations grouped by type.
parameters:
type: object
properties:
outputLevel:
type: string
description: Supports condensed or complaints
container:
image: vonwig/read_eslint
Now we can use this tool to finish our prompt.
You are an AI assistant who specializes in resolving lint violations in projects. Use the tools available to quickly take action and be very brief.
1. Run lint. // Will write /thread/eslint.json
2. Evaluate total violations. // Will read /thread/eslint.json
<10 violations: Parse output with complaints output.
10+ violations: Parse output with condensed output.
3. Fix the violations using the following steps:
## Condensed:
{>fixing}
A user prompt of:
1. Get the line Use read_eslint with the `path` arg to get all of the violations for a file.
2. Make the correction Respond in the following format:
```json
{
"start": [1, 4],
"end": [2, 4],
"edit": "Lorem ipsum"
}
```
Once you have fixed one file, move on to the next.
You are able to fix the following violations:
no-async-promise-executor
...
yields:
The violation is located in the file src/utils/promiseUtils.ts at line 30-31. The current code is:
```javascript
return new Promise(async (resolve, reject) => {
...
});
```
To resolve the no-async-promise-executor violation, remove the async keyword from the `Promise` executor function. The corrected code would look like:
```javascript
return new Promise((resolve, reject) => {
...
});
```
Unfortunately, we’re still not seeing a good result. Without the violating context from the code, again, our assistant can’t provide an actionable fix. So we need to make a tool to get that.
Tree Sitter
Without the right tools, the LLM is, at best, vague and helpful — and at worse quite liable to hallucinate. We need tools to let the LLM be actionable here. Tree-sitter is a popular open source tool we can use to parse a variety of languages into abstract syntax trees (or ASTs).
The powerful part of Tree-sitter that we want to take advantage of is the community of grammar authors providing support for almost any language our assistant could encounter. We have Dockerized this tool to get context given a file path and line number.
Results
We have trimmed debug finish responses from the payload but kept function calls.
## ROLE assistant
...({:function {:name run_lint, :arguments {}}, :id call_MaK7CGpOISktz1fZP3GS7m4H})
## ROLE assistant
.........................({:function {:name run-eslint, :arguments {
"args": ".",
"version": 8,
"outputLevel": "summary"
}}, :id call_RHX2hztWvghxbyZx5cvSXgMU})
## ROLE tool (run-eslint)
Running npx with args: eslint@8 --format json .
Peparing output
Found 41 violations in 271 files.
## ROLE tool (run_lint)
I ran ESLint (version 8) on your entire project. Here are the results:
- 271 files were checked for violations
- Out of these, 41 files had one or more violations
## ROLE assistant
.............({:function {:name parse_lint_results, :arguments {
"outputLevel": "condensed"
}}, :id call_sYHP8tXuKbXGaXdNCdO3W5kU})
## ROLE tool (parse_lint_results)
{ "@typescript-eslint/no-floating-promises": [ "src/commands/help.ts",
"src/commands/registerWorkspaceCommand.ts" ], "@typescript-eslint/no-explicit-any": [ "src/commands/registerLocalCommand.ts" ], "@typescript-eslint/naming-convention": [ "src/constants.ts" ], "eqeqeq": [ "src/debugging/DockerDebugConfigurationProvider.ts", "src/debugging/DockerServerReadyAction.ts", "src/runtimes/files/ContainerFilesProvider.ts", "src/tree/LocalRootTreeItemBase.ts" ], "@typescript-eslint/no-unused-vars": [ "src/extension.ts", "src/scaffolding/wizard/netCore/NetCoreGatherInformationStep.ts" ], "@typescript-eslint/semi": [ "src/utils/registryExperience.ts" ] }
## ROLE assistant
...............({:function {:name violations_for_file, :arguments {
"path": "src/commands/help.ts"
}}, :id call_fAN5kTrXndaNs7Ix4iH51E4T})
## ROLE tool (violations_for_file)
Getting eslint.json for path: src/commands/help.ts
ESLint violations for src/commands/help.ts:
{ "filePath": "src/commands/help.ts", "messages": [ { "ruleId": "@typescript-eslint/no-floating-promises", "severity": 1, "message": "Promises must be awaited, end with a call to .catch, end with a call to .then with a rejection handler or be explicitly marked as ignored with the void operator.", "line": 35, "column": 5, "nodeType": "ExpressionStatement", "messageId": "floatingVoid", "endLine": 35, "endColumn": 100] }
## ROLE assistant
......................({:function {:name run_tree_sitter, :arguments {
"path": "src/commands/help.ts",
"line": 35
}}, :id call_cKnNRQs5hmMsTszTfDnwfK12})
## ROLE tool (run_tree_sitter)
{
offending_line: '}',
line_node: SyntaxNode {
type: },
startPosition: {row: 35, column: 0},
endPosition: {row: 35, column: 1},
childCount: 0,
},
parent: SetNode {
type: set,
startPosition: {row: 33, column: 45},
endPosition: {row: 35, column: 1},
childCount: 4,
},
parent_text: '{\n' +
" vscode.env.openExternal(vscode.Uri.parse('https://github.com/microsoft/vscode-docker/issues'));\n" +
'}'
}
## ROLE assistant
The violation on line 35 of the `src/commands/help.ts` file is due to a floating promise. To resolve this, add the void operator to the beginning of the line as below:
```js
void vscode.env.openExternal(vscode.Uri.parse('https://github.com/microsoft/vscode-docker/issues'));
```.
This result demonstrates that the LLM can orchestrate a small code improvement using the right tools.
Failures
Unfortunately, it terminates after coming to the end of the first violation rather than continuing on to the next. This seems to be difficult to change, even trying two other combinations of prompts failed to get a loop going.
Fix any lint violations you can using the tools provided.
After resolving one violation, move onto the next.
Once you have fixed one file, move on to the next.
Successful prompt
However, what we did find was that specifying the lint violation to focus on did work:
Please only fix no-unused-vars
yields:
## ROLE assistant
The violation in the file "src/extension.ts" is a declared variable 'unuseVar' at line 41, which is not used anywhere. The fix would be the removal of that line..nil
Figure 3 lists a few interesting violations we tested.
Overall, we can see that our assistant, when given the right tools to find and fix violations, can make good code change suggestions for even difficult lint violations. We welcome you to check out the code and test with different violations that we didn’t cover.
Summary
We continue to investigate the use of tools for gathering context and improving suggestions. In this installment, we have looked at how AI assistants can provide significant value to developers by:
- Using a linter to learn about violations in a TS project without Node, NPM, or ESLint.
- Leveraging an ephemeral volume to store large context without using tokens.
- Using Tree_sitter to load precise, relevant code context from a project.
- Generating fixes for violations in a TS project.
As always, feel free to follow along in our new public repo and please reach out. Everything we’ve discussed in this blog post is available for you to try out on your own projects.
Learn more
- Subscribe to the Docker Newsletter.
- Read the Docker Labs GenAI series.
- Get the latest release of Docker Desktop.
- Have questions? The Docker community is here to help.
- New to Docker? Get started.