`"module": "node16"` should support extension rewriting

[arendjr]

Bug Report

TypeScript 4.7 RC introduces the "module": "node16" setting. When used, this requires users to use the .js extension in import paths. I strongly believe this to be a mistake and TypeScript should instead allow users to use the .ts/.tsx extension of the source file. It would then be the TS compiler’s responsibility to rewrite this to the output file as necessary.

I have several reasons for requesting this change:

• The file with the .js extension most likely does not exist on disk. It may exist on disk if a build has been created and the output is placed right alongside the sources (not a common configuration), but otherwise the file is simply not there or in a different place than one might expect. This is confusing to users, because they need to manually consider how output files are created.
• Using the .js extension makes the source code less portable for full-stack projects. If a user wants to use the same sources for a frontend project that uses a bundler, they will be out of luck. A bundler would be able to understand a reference to the .ts file (because that one actually exists on disk!), but would struggle with references to files that do not exist for its configuration.
• Using the .js extension makes the source incompatible with Deno projects, as they use the .ts/.tsx extensions, because those are the ones that actually exist.
• Using the .js extension is inconsistent with type import. Does import type { MyType } from “./myFile.js” even work? It would not make sense because the JS file contains no types. But if it doesn’t work, does that mean I still have to use the .ts extensions only for the types? That would be highly annoying.

🔎 Search Terms

extension rewriting esm module

🕗 Version & Regression Information

TypeScript 4.7. RC1

• I was unable to test this on prior versions because the feature is new to version 4.7.

🙁 Actual behavior

I am forced to write import { myFunction, type MyType } from “./myFile.js” despite “./myFile.js” not being a valid path in my project.

🙂 Expected behavior

I should be able to write import { myFunction, type MyType } from “./myFile.ts” because “./myFile.ts” is the actual file I am referring to. Upon compilation I would expect TypeScript to rewrite the path so the output works correctly too.

[RyanCavanaugh]

We don't rewrite target-legal JavaScript code to be different JavaScript code from what it started with, no matter the configuration.

See also #16577 (comment)

[arendjr]

I’m sorry, but I cannot understand why you would label this issue as Out of scope or why you would dismiss this based on a comment from almost one and a half year ago.

You argue “we don’t rewrite target-legal JavaScript code”. But this argument does not apply. If someone wrote import { myFunction, type MyType } from “./myFile.ts”, which I believe to be a very reasonable expectation from a user’s point of view, and you would emit that with the “.ts” extension intact, the resulting code would be invalid because that target doesn’t exist. You are building the compiler, you know that target doesn’t exist. So you wouldn’t be rewriting valid code, you would be rewriting code you know would be invalid if you didn’t rewrite it. I hope you’re not arguing it’s better to let the user shoot themselves in the foot.

Furthermore, the comment you linked depends on the arguments presented in yet another comment: #16577 (comment)

If you read the arguments there, this is clearly a different situation than what we are facing today. My issue is with the change in the new module resolution for ESM introduced in TypeScript 4.7. This change forces users to specify the .js extension. Compare this to the “facts” presented in the comment I linked, which clearly apply to a situation in which the user is free to choose whether they want to include the extension or not. This is not the situation as it is with the ESM resolution in TypeScript 4.7.

Back then, users could improve compatibility by adding the extension, but the way they are now forced to use it compromises compatibility as I attempted to explain, but none of which you addressed, and none of which any of the linked comments address.

As such I would kindly ask you to reconsider the implications of the new module resolution, including the impact on developers and the wider ecosystem.

[RyanCavanaugh]

I don't think there's anything new to discuss here; the strong line we've drawn in the sand is that we do not rewrite import paths, ever. Our design principle, consistently, is that you write the import path that you want to appear in your JS file, and configure TS to tell it how to turn that path into a build-time path. Doing it differently is out of scope for our project.

This is a hard constraint per design goal number 7, "Preserve runtime behavior of all JavaScript code.". import { foo } from "./x" is JavaScript code. Granted import { foo, type bar } ... is not, but I think it's clearly way too confusing to have one runtime path appear if you put the type keyword on a single named import and a different path if you don't.

Nothing about node16 resolution changes this. Our principle even would be the same if some new module resolution system appeared where you wrote the sha256 hash of the file you wanted -- we would have you write import foo from "ba932c1fd3..." and then have a system to map that to foo.ts, not have you wrote import foo from "foo.ts" and then emit some JS other than what you wrote.

If you have some environment where you sometimes need to have JS that says import foo from "blah.js" and sometimes need to have JS that says import foo from "blah", that is also out of scope for our project, the same as if you needed the same input line of TypeScript to sometimes emit 3 * 4 and sometimes emit 3 + 4. There are tools to handle these problems and it's out of scope for us to re-implement solutions to those problems.

There are surely difficulties in navigating the module ecosystem, no one can deny that. What I'm saying is, rewriting import paths is not a solution we consider to be in-scope. We're always interesting in hearing ideas that can simplify the module resolution process and improve the user experience as long as it fits within our design criteria of not rewriting import paths.

[arendjr]

Thanks for your reply. I understand I’m getting at some pretty fundamental issues here, so I try to navigate this as delicately as I can.

First, the new thing I mentioned is the node16 module resolution mechanism that comes with TypeScript 4.7. Unlike previous module resolutions supported by Typescript, this forces users to use the .js extension in their paths. While this may be a consequence of decisions that were made earlier, this does exacerbate the issues that result from those decisions. If nothing else, I believe that may present a valuable moment to reflect on those decisions to see if they still achieve the goals you set out to achieve.

I also believe the timing is important here. So far, and as far as I’m aware, Typescript users have mostly avoided including extensions in their paths. And for good reason: Including them makes their code less compatible with existing bundlers used in the ecosystem. One notable exception here are Deno users, who include the .ts extension, in a resolution mechanism that follows the ESM resolution in spirit.

If TypeScript 4.7 is released with the node16 setting as it’s implemented today, I’m afraid this may introduce further schism between the various user groups. This is unfortunate, because node16 is presented as the way to do ESM with TypeScript, but for users to use it they will be forced to move away from their current practices in a manner that is incompatible with their current tools. It will become harder for a single codebase to support Node.js and bundlers, while there is no technical reason for this to be so.

So while there’s no technical reason for this outcome you admit to be unfortunate, you do present one based on principle: You say you “do not rewrite import paths, ever” and back this up with the design goal that says to "Preserve runtime behavior of all JavaScript code."

So let’s reflect on this. Because if these principles/design goals will inevitably lead us to face such an unfortunate outcome, is it possible the design goal itself is in need for reconsideration? I would not necessarily go that far, but I do think your proverbial line in the sand to “not rewrite import paths, ever” is an unnecessarily strict interpretation of the design goal.

Let’s revisit my original request, and let’s simplify it to focus on the core issue:

import { myFunction } from “./myFile.ts”

Is this valid JavaScript code? It’s certainly syntactically valid. But if “./myFile.ts” does not exist or contains syntax that is not valid JS, it’s certainly not functional JavaScript code. Does it make sense to wish to preserve behavior of non-functional code?

But let’s take it a step further: How is this code supposed to behave? Again I will refer to Deno, because it defines actual runtime behavior for this that is inline with the ESM specification in spirit. But here’s the rub: Deno provides a TypeScript runtime, not merely a JavaScript one. So this code is valid TypeScript code, even if it is not functional JavaScript code.

Let’s look at that design goal one more time: "Preserve runtime behavior of all JavaScript code."

The behavior is clear. Just run it in Deno and see. But interestingly, it is the TypeScript code emit that breaks this behavior! Because after the code emit, the file it referred to is no longer there.

I know you will defend this by saying the path is intended to be an output path, but how does that follow from your design goal to not break behavior? If anything, this intention feels more like a post hoc rationalization to defend the “do not rewrite import paths, ever” position than it seems a useful part of the design. It’s certainly not a design goal.

Now, I realize there are no easy answers here. But from my perspective, the design goal to "Preserve runtime behavior of all JavaScript code" would be served by rewriting import paths if the behavior would otherwise be broken by your own emit process.

The statement import { myFunction } from “./myFile.ts” is valid TypeScript (even functional JavaScript under very strict circumstances) whose behavior would be broken if you do not rewrite the path during its emit phase.

While it is your prerogative to define the scope of your project, in my opinion it is a stretch to claim an issue introduced by your own process (it’s inevitable that paths break when you write files to a new location with new names) as out of scope.

Finally, I would like to come back to the hypothetical example you raised:

Our principle even would be the same if some new module resolution system appeared where you wrote the sha256 hash of the file you wanted -- we would have you write import foo from "ba932c1fd3..." and then have a system to map that to foo.ts, not have you wrote import foo from "foo.ts" and then emit some JS other than what you wrote.

What I find interesting here is that you say you’re willing to build a system to do the mapping, but you’re not willing to help the user writing the code. It seems you hold stronger to the notion of not rewriting import paths than to preservation of behavior or the notion of creating value for your users. We both have the same design goal in mind (to maintain the behavior of the code), but to me this shows your “do not rewrite import paths, ever” position is untenable and it feels not in line with what I believe would be the best interests of the project.

I hope I have not offended with this post. I raised this issue because I believe it would be harmful to the TypeScript community if segments of the community have to use incompatible module resolution schemes, which the node16 resolution as it stands further forces users towards. Instead, I hope that through a relatively simple change we can make people’s code more easily interoperable. Even if that change means a more fundamental reinterpretation of the original design goals.

[arendjr]

As I fear this conversation might otherwise go entirely stale, I would like to offer one more argument as to why I think the “do not rewrite import paths, ever” position is not just undesirable, but actually runs counter to your own design goals.

Let’s look at another variation of the example we discussed so far:

import { myFunction, MyType } from “./myFile.ts”

Let’s assume MyType is not a class, but a TS type that gets erased during emit. What would the output be? It would become:

import { myFunction } from “./myFile.ts”

That’s interesting, isn’t it? The original snippet was 100% syntactically correct JavaScript. And yet it was rewritten as part of the emit process.

Of course rewriting it was the correct thing to do. Without rewriting, the emit process would have yielded output the JavaScript runtime would have trouble resolving. So we must rewrite JavaScript in order to preserve behavior.

Please explain to me how the import path is different. Sticking to “do not rewrite import paths, ever” is leading to output the JavaScript runtime will have trouble resolving — the exact same behavioral issues that you are willing to solve elsewhere. Issues you must solve in order to adhere to your own design goal of preserving behavior.

Of course you could argue that MyType refers to a TypeScript type, so you actually rewrote TypeScript rather than JavaScript. But in that case, please explain to me how the import path is different. “./myFile.ts” refers to a TypeScript file, so you would rewrite TypeScript rather than JavaScript.

Rewriting paths to TypeScript files in order to preserve behavior is compatible with your original design goal. To “not rewrite import paths, ever” is not.

[mlandalv]

It feels like many of these problems would be solved if TS enforced the use of file extensions in import/export paths. Not .js but the actual extension, eg .ts or .tsx.

Then it's up to the transpilation process to output code compliant with the specified module and target. If that means changing .ts to .js then so be it.

(What if tsc eventually gets support for a --out-file-extension=.mjs flag. Then those .js in the paths won't work anymore.)

[s123121]

Technical aside, this issue make the DX worse for everyone. It doesn't make any sense for relative import to refer to a non-existent .js file. On the top of my hat, to circumvent this problem, the developer need to:

• Change eslint setting for missing import
• Some tools need to reconfigured to resolve non-existent resource (like Deno, ts-loader, ...)
• vscode can navigate between files just fine but what about other text editor

[neil-morrison44]

I think some of the issue stems from this

is that you write the import path that you want to appear in your JS file, and configure TS to tell it how to turn that path into a build-time path.

as a user I see the typescript compiler taking valid TS code from the “src” realm and converting it to valid JS code in the “build” realm.

When I first heard about the “.js” extension it broke what I understood to be the separation between “src” & “build” - I wondered if I’d need to supply the full path to the output .js in the source file, or otherwise adjust it based on my knowledge of where the .js file would end up

[calebpitan]

If only there was a way to put this to vote!

@RyanCavanaugh’s argument, which I believe represents that of the TypeScript team, doesn’t seem sustainable on the long run. If a community of other engineers says this is what they think is best, then it better be given a strong consideration, rather than being labeled “out of scope” just for the sake of wanting to be conservative. Except TypeScript was made only for the TypeScript team.

The system is changing! If need be that some principles are reviewed as the system changes, to better support and adapt these changes then so be it. Otherwise the system shouldn’t change and fundamental principles be kept.

Moreover, @arendjr’s arguments in summary has not even changed the principles it has only broadened the short-sighted interpretations and undue strictness of these principles.

Oh and I think it is being put to vote already seeing how many reactions are yet in favor of @arendjr’s opinion.

[fabis94]

I hope the TS team reevaluates their stance, the points made in the OP definitely make a lot of sense