Skip to content

Load: Optimized the downgraded logic for tsFile to insert more data when tsFile corrupted#17674

Open
Caideyipi wants to merge 2 commits into
masterfrom
load-query
Open

Load: Optimized the downgraded logic for tsFile to insert more data when tsFile corrupted#17674
Caideyipi wants to merge 2 commits into
masterfrom
load-query

Conversation

@Caideyipi
Copy link
Copy Markdown
Collaborator

Description

As the title said.


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

@sonarqubecloud
Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Copy link
Copy Markdown
Member

@luoluoyuyu luoluoyuyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Introduces LoadTreeTsFileTabletIterator and uses it in LoadTreeStatementDataTypeConvertExecutionVisitor so corrupted TsFiles can yield more insertable tablets on downgrade. Adds unit tests for the converter path.

Please fix failing datanode unit tests before merge.

null,
true)) {
for (final Pair<Tablet, Boolean> tabletWithIsAligned : parser.toTabletWithIsAligneds()) {
try (final LoadTreeTsFileTabletIterator tabletIterator =
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replacing TsFileInsertionEventScanParser with LoadTreeTsFileTabletIterator centralizes corrupted-file handling for load. Please document what data may still be skipped versus inserted when a file is partially corrupted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants