Hudi's Lazy Writes: Rollback & Clean Before Commit

Jan 16, 2026 by Editorial Team 51 views

Hey everyone! Let's dive into a cool feature for Apache Hudi that's all about making those pesky write failures a little less painful. We're talking about giving users more control over how Hudi handles failed writes, specifically by adding options to automatically rollback or clean up before a commit. This is super important because, as we've seen, failed writes can lead to a messy situation in your data lake, causing all sorts of problems. So, let's break down why this matters, what we're doing, and how it helps!

The Problem: Failed Writes and Data Clutter

Okay, imagine this: You've got a job ingesting data into Hudi. Everything's humming along, and then bam—something goes wrong. Maybe a network hiccup, a Spark job times out, or some other unexpected issue pops up. Your write job fails. If you're using Hudi with a LAZY rollback policy, those failed writes aren't automatically cleaned up between job runs. This means files from those failed attempts stay hanging around in your data lake. Over time, these files start piling up, taking up precious storage space and potentially bumping into file quota limits. That’s a headache, right?

This is where the new functionality comes in. The goal is to give users more control over how Hudi handles these scenarios. By proactively rolling back or cleaning up before a new commit, we can prevent the buildup of these unwanted files. It's like having a cleanup crew ready to sweep up the mess before the next data load. This ensures your data lake stays clean, efficient, and avoids those nasty quota issues. We will be discussing ways to configure Hudi to address these problems.

Think of it like this: your data lake is your house. You don't want to keep piling up trash and clutter without ever cleaning it up, because sooner or later, you won't have any space to move around! Hudi's new features are like having a regular cleaning schedule for your data, keeping everything tidy and running smoothly. By automatically taking care of failed writes, we reduce the chances of running out of space and reduce the chance of errors.

The Solution: New Policies and Clean Configuration

So, what's the plan to tackle this problem? We're introducing a couple of new features to give you more control:

LAZY_WITH_PREWRITE Policy: This is like an enhanced version of the existing LAZY policy. With LAZY_WITH_PREWRITE, Hudi will try to rollbackFailedWrites during the startCommit process, before even attempting to create a new instant. This means that before a new write starts, Hudi will first attempt to clean up any files left over from failed writes. It's all about preventing that clutter from building up in the first place.
Clean Configuration in startCommit: We're also adding a new configuration option that, when enabled, tells Hudi to run a clean operation during startCommit. Think of this as a proactive measure. Before a new instant is created, Hudi will attempt to clean up any unwanted files. This can be especially useful in cases where the write job might have completed the commit but fails before the post-commit clean process kicks in, which could cause a similar buildup of files.

These additions are all about giving users flexibility. With these new options, you can choose how aggressive you want Hudi to be in cleaning up failed writes. If you're dealing with a lot of failed jobs, you might opt for the LAZY_WITH_PREWRITE policy. Or, if you want a more general cleanup before each commit, the clean configuration option could be the better fit. The idea is to make sure your data lake stays clean, even when things go wrong.

This approach not only prevents storage issues but also ensures that your data is always in a consistent state. It reduces the risk of having incomplete or corrupted data in your lake. By proactively cleaning up before the next commit, you're essentially guaranteeing that only valid and consistent data will be part of the new instant. This is a game-changer for those dealing with frequent write failures.

Why This Matters: Benefits and Use Cases

Why should you care about this? Well, here are a few reasons why this new functionality is a big deal:

Preventing Storage Issues: As mentioned, the main benefit is preventing the accumulation of unwanted files. This helps avoid running into storage quota limits and keeps your data lake running smoothly. It is like regularly taking out the trash.
Improved Data Consistency: By ensuring that failed writes are handled before new commits, we improve the consistency of your data. You'll have fewer issues with incomplete or corrupted data.
Reduced Manual Intervention: With these automated cleanup options, you'll need to spend less time manually cleaning up after failed writes. This frees up your time to focus on other important tasks.
Enhanced Reliability: By proactively addressing potential issues, you can improve the overall reliability of your data pipeline. Fewer errors mean fewer headaches.

Use Cases

Let’s look at some examples of where this is helpful:

Frequent Job Failures: If your ingestion jobs are failing often, either due to infrastructure issues or data quality problems, the LAZY_WITH_PREWRITE policy is your friend. It ensures that files from failed writes are cleaned up before the next attempt.
Post-Commit Failure Scenarios: If your write jobs are completing the commit, but fail during post-commit steps (like cleaning), the clean configuration option will take care of the files.
Environments with Strict Quotas: If you're running in an environment with tight storage quotas, these options will help you stay within those limits.

Imagine you are using a Hudi data lake for your business. Let's say you have a critical process for customer analytics. This process runs daily. On some days, there might be errors from third-party APIs. Or network issues, and write jobs will fail. Without these new features, these failures could result in lots of junk data. And eventually, they could interrupt your daily analytics. With the LAZY_WITH_PREWRITE or clean configuration, your Hudi data lake ensures that data is always consistent, reliable, and up to date, providing you with the accurate insights needed for decision-making.

Implementation Details and Upstreaming

We’ve already added this functionality to our internal 0.x Hudi build, and we're planning to upstream it. This means that we're working on integrating this code into the main Hudi codebase. Before we can do that, we need to make sure everything is in place and that the community agrees on the approach. Upstreaming will make these features available to everyone, allowing more people to benefit from the improved write handling.

Here are the key steps involved:

Code Review: The code will undergo a thorough review by other Hudi contributors to ensure it meets the project's standards. This helps identify any issues and ensures the code is well-written and maintainable.
Community Discussion: We'll discuss the proposed changes with the Hudi community to gather feedback and make any necessary adjustments. This ensures everyone is on board with the new features.
Testing: Extensive testing will be performed to make sure the new features work as expected and don't introduce any regressions.
Integration: After all the checks, the code will be integrated into the main Hudi codebase, making the new features available to everyone.

Conclusion: A Cleaner, More Reliable Hudi

In a nutshell, we're making Hudi even more robust by giving you better tools to handle those inevitable write failures. By proactively rolling back or cleaning up before commits, you can avoid storage issues, improve data consistency, and reduce manual intervention. This helps make Hudi even more reliable. It's about making sure your data lake stays clean, efficient, and ready to handle all your data needs. This ensures your data is consistently updated, reliable, and free from clutter. With these new features, you can have confidence in your data pipeline, knowing that Hudi is working hard behind the scenes to keep things running smoothly. So, stay tuned for the official release, and get ready to enjoy a cleaner, more reliable Hudi experience!