Skip to content

Make tests engine agnostic#16272

Closed
Trott wants to merge 15 commits into
nodejs:masterfrom
Trott:engine-agnostic
Closed

Make tests engine agnostic#16272
Trott wants to merge 15 commits into
nodejs:masterfrom
Trott:engine-agnostic

Conversation

@Trott
Copy link
Copy Markdown
Member

@Trott Trott commented Oct 18, 2017

We treat error message changes as breaking changes and therefore often test for exact error messages. However, this only applies to errors generated by Node.js core. For errors generated by V8, the TSC has determined that we do not consider it a breaking change when an error message changes.

We have a relatively easy way to find these errors in our tests because ChakraCore will often (usually? almost always?) need to modify such tests to match their error messages.

This PR changes several tests that node-chakracore needed to modify due to the overzealous error message checking (much of which I may have been responsible for either directly or indirectly, but hey, we all get to learn from our mistakes).

There are still more tests to be done, but this is a good start and reaction to this will inform the approach in the rest of the tests.

@nodejs/chakracore @nodejs/testing

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • commit message follows commit guidelines
Affected core subsystem(s)

test

@Trott Trott added the test Issues and PRs related to the tests. label Oct 18, 2017
@Trott
Copy link
Copy Markdown
Member Author

Trott commented Oct 18, 2017

Copy link
Copy Markdown
Contributor

@digitalinfinity digitalinfinity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome- thanks for this!

Copy link
Copy Markdown
Contributor

@boingoing boingoing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great!

💯 👍

Comment thread test/parallel/test-cli-syntax.js Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: JavaScript engine

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@targos Fixed!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the best we can do?
Can we at least test that it's a multiline string?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@refack I guess that's something. I've added this:

  // Test that it's a multiline string.
  assert.ok(/\n/g.test(stack));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

Copy link
Copy Markdown
Member

@hiroppy hiroppy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM🍻

Copy link
Copy Markdown
Member

@joyeecheung joyeecheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, although I wouldn't even mention "V8 and ChakraCore", just say "different JavaScript engines"...

Comment thread test/parallel/test-error-reporting.js Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: should be "At least one line ... " or the assertion should be > 1.

Copy link
Copy Markdown
Contributor

@cjihrig cjihrig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I agree with @joyeecheung that mentioning the engines by name is not necessary.

Comment thread test/parallel/test-console-count.js Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Trott, I know it's a ton of tedious work, but can these assert.throws() be modified to common.expectsError() for sake of consistency?

e.g.

common.expectsError(
  () => console.count(Symbol('test')),
  { type: TypeError }
);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(feel free to say no for doing so in this PR :-) ...)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like a great code-and-learn PR

Copy link
Copy Markdown
Member

@trivikr trivikr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gireeshpunathil
Copy link
Copy Markdown
Member

@Trott
Forgive me for raising voice on a PR that is the result of a great work, but thought I should mention this here. In an effort to resolve the vm-neutrality gaps, we seem to address it at it's manifestation stage, not at the root - with a side effect of relaxing error checks to their most abstract form. TypeError by far is the most common error in Javascript, and stripping the exact details at error sites may cause some of the real bugs to escape the tests?

Given that

TSC has determined that we do not consider it a breaking change when an error message changes.

Should we work with v8 and chakra to unify their error messages?