Load: Optimized the downgraded logic for tsFile to insert more data when tsFile corrupted#17674
Load: Optimized the downgraded logic for tsFile to insert more data when tsFile corrupted#17674Caideyipi wants to merge 2 commits into
Conversation
|
luoluoyuyu
left a comment
There was a problem hiding this comment.
Review summary
Introduces LoadTreeTsFileTabletIterator and uses it in LoadTreeStatementDataTypeConvertExecutionVisitor so corrupted TsFiles can yield more insertable tablets on downgrade. Adds unit tests for the converter path.
Please fix failing datanode unit tests before merge.
| null, | ||
| true)) { | ||
| for (final Pair<Tablet, Boolean> tabletWithIsAligned : parser.toTabletWithIsAligneds()) { | ||
| try (final LoadTreeTsFileTabletIterator tabletIterator = |
There was a problem hiding this comment.
Replacing TsFileInsertionEventScanParser with LoadTreeTsFileTabletIterator centralizes corrupted-file handling for load. Please document what data may still be skipped versus inserted when a file is partially corrupted.




Description
As the title said.
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR