Build A Document Processing Service With NestJS

Jan 20, 2026 by Editorial Team 48 views

Build a Robust Document Processing Service with NestJS

Hey there, code enthusiasts! Ever dreamt of building a powerful document processing service? Well, you're in the right place! We're diving deep into creating a Document Processing Service using the amazing NestJS framework. This project is designed for handling land ownership documents, but the principles and techniques you'll learn are super transferable to any document-heavy application. We'll be covering document upload, Optical Character Recognition (OCR) processing, metadata extraction, and text analysis. Let's get started!

Project Overview: The Document Processing Service

Our Document Processing Service will be a self-contained NestJS module, packed with features to handle all things document-related. This module will be able to ingest PDF and image files, use OCR to extract text, parse important metadata, store the extracted data, and provide mechanisms for tracking progress and handling errors. We will be using a queue-based system to handle large documents. This architecture ensures smooth operations even when dealing with massive files. This project is perfect for anyone looking to build a document management system, automate data extraction, or simply expand their skills with the NestJS framework. So, get ready to roll up your sleeves and dive into the exciting world of document processing!

Core Components: Unpacking the Requirements

Let's break down the essential components that will make up our Document Processing Service: We'll begin by constructing a DocumentProcessingModule containing services, controllers, and entities. This structure will provide an organized, maintainable foundation for the project. The first function will enable support for PDF and image uploads, so it can handle a wide array of document formats. Then we will integrate libraries like PDF.js and Tesseract.js to get it working. The next part will be the extraction of text content using Optical Character Recognition (OCR), which is crucial for digitizing the document contents. Then we can go on and parse the metadata such as dates, names, parcel IDs, and coordinates. This step requires careful parsing techniques and the creation of entities to store structured data. We'll then set up database integration with PostgreSQL, where extracted data will be securely stored. We'll also implement a queue-based processing system, using Bull to handle large documents. This prevents bottlenecks and ensures the system remains responsive. Progress tracking for asynchronous operations will be added as well, so users can monitor the status of their documents. Finally, we'll implement robust error handling and retry logic, which guarantees that issues are addressed gracefully, and the system can recover from failures.

Technology Stack: The Tools of the Trade

To make this vision a reality, we'll be using a robust and versatile tech stack. Our foundation will be NestJS, a powerful framework for building efficient and scalable server-side applications. We will integrate TypeORM for object-relational mapping (ORM), simplifying database interactions. PostgreSQL will be our database of choice, offering reliability and scalability. For handling asynchronous tasks and queue management, we'll be using Bull, a robust queue library. Tesseract.js will be used for OCR processing, making text extraction a breeze. Last but not least, we will be using PDF.js for PDF parsing and handling.

Building the Document Processing Module: Step-by-Step

Alright, let's get into the nitty-gritty of building our Document Processing Module. We'll walk through the process step-by-step, ensuring that you grasp every detail. From setting up the module to implementing each feature, we'll cover it all. You'll not only learn how to build the service but also understand the underlying concepts that make it work. Get ready to turn your ideas into a fully functional document processing service!

Setting Up the NestJS Module

First things first, let's set up our NestJS module. We'll start by creating a new NestJS project. You can use the Nest CLI to generate a new project. Then, we will create the DocumentProcessingModule. Inside the module, we'll declare the necessary providers: services, controllers, and any other dependencies. This modular structure keeps our code organized and manageable. We'll then configure the module to use the required dependencies, such as the database connection and the queue system.

npm install -g @nestjs/cli
nest new document-processing-service
cd document-processing-service
nest g module document-processing

Document Upload and File Handling

Now, let's handle document uploads. We'll build an API endpoint that accepts PDF and image files. We'll use libraries like multer to handle file uploads. Make sure to include file validation and sanitization. This is very important for security and to prevent any issues with processing malformed files. We'll need to specify the allowed file types and sizes to prevent malicious uploads. After uploading, the files will be stored in a temporary location, ready for processing.

import { Controller, Post, UseInterceptors, UploadedFile, ParseFilePipe, MaxFileSizeValidator, FileTypeValidator } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { DocumentProcessingService } from './document-processing.service';

@Controller('documents')
export class DocumentProcessingController {
  constructor(private readonly documentProcessingService: DocumentProcessingService) {}

  @Post('upload')
  @UseInterceptors(FileInterceptor('file'))
  async uploadFile(
    @UploadedFile(
      new ParseFilePipe({
        validators: [
          new MaxFileSizeValidator({ maxSize: 10 * 1024 * 1024 }), // 10MB
          new FileTypeValidator({ fileType: '.(pdf|png|jpeg|jpg)' }),
        ],
      }),
    )
    file: Express.Multer.File,
  ) {
    return this.documentProcessingService.processDocument(file);
  }
}

Implementing OCR with Tesseract.js

Next up, Optical Character Recognition (OCR) with Tesseract.js! This is where the magic happens. We'll integrate Tesseract.js into our service to extract text from images and PDFs. The basic flow involves:

Loading the image or PDF.
Passing it to Tesseract.js for processing.
Receiving the extracted text. We need to handle different languages and improve OCR accuracy by adjusting image preprocessing techniques (e.g., noise reduction, contrast enhancement). We should also handle potential OCR errors gracefully. You can enhance the accuracy of text extraction by preprocessing images. This involves adjusting contrast, removing noise, and improving the image's clarity, which is essential for accurate OCR.

import { Injectable } from '@nestjs/common';
import * as Tesseract from 'tesseract.js';

@Injectable()
export class DocumentProcessingService {
  async extractText(file: Express.Multer.File): Promise<string> {
    try {
      const result = await Tesseract.recognize(file.buffer, 'eng'); // Adjust language as needed
      return result.data.text;
    } catch (error) {
      console.error('OCR Error:', error);
      throw new Error('Failed to perform OCR.');
    }
  }
}

Metadata Extraction and Parsing

Extracting metadata is another essential step. After the text has been extracted from the documents, we will need to analyze the text to extract relevant metadata like dates, names, parcel IDs, and coordinates. This is going to involve using regular expressions, string parsing, and possibly some machine-learning techniques for advanced parsing. We'll create a structured object to store the extracted metadata. This structured format will make data querying and analysis straightforward. Implement error handling to manage cases where metadata cannot be parsed correctly. This might include logging errors, setting default values, or notifying administrators.

// Example metadata extraction using regex
function extractMetadata(text: string) {
  const dateRegex = /(\d{2}[/-]\d{2}[/-]\d{4})/g; // Matches dates like 01/01/2023 or 01-01-2023
  const parcelIdRegex = /\b(PARCEL-\d+)\b/g; // Matches parcel IDs like PARCEL-123

  const dates = text.match(dateRegex) || [];
  const parcelIds = text.match(parcelIdRegex) || [];

  return {
    dates,
    parcelIds,
  };
}

Database Integration with PostgreSQL

We will use TypeORM to integrate with the PostgreSQL database. First, define the entities that match the data we want to store, such as metadata and file information. Then, configure TypeORM to connect to your PostgreSQL database. Next, use repositories to perform database operations, such as saving metadata, updating document statuses, and fetching data. Test the database integration thoroughly. Make sure data is stored and retrieved correctly. Also, remember to handle database connection errors.

import { Entity, Column, PrimaryGeneratedColumn } from 'typeorm';

@Entity()
export class Document {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  filename: string;

  @Column()
  extractedText: string;

  @Column({ type: 'jsonb', nullable: true })
  metadata: object;

  @Column({ default: 'processing' })
  status: string;
}

Queue-Based Processing with Bull

To handle large documents and prevent blocking, we will use Bull. We'll configure Bull with Redis as the message queue provider. Then, we will create a queue and add jobs for document processing. This involves the uploading, OCR, metadata extraction, and data storage. Process the jobs asynchronously by defining a worker process. This worker will execute the main tasks, such as OCR and database updates. Use progress tracking to monitor the status of each job. Also, add retry mechanisms to handle any failures during processing. This ensures that documents are processed reliably.

import { Processor, Process } from '@nestjs/bull';
import { Job } from 'bull';
import { DocumentProcessingService } from './document-processing.service';

@Processor('document-processing')
export class DocumentProcessor {
  constructor(private readonly documentProcessingService: DocumentProcessingService) {}

  @Process('processDocument')
  async handleProcessDocument(job: Job<any>) {
    try {
      const { file } = job.data;
      const extractedText = await this.documentProcessingService.extractText(file);
      const metadata = extractMetadata(extractedText);

      // Save to database using your service
      await this.documentProcessingService.saveDocumentData(file.originalname, extractedText, metadata);

      job.progress(100);
      return { status: 'completed' };
    } catch (error) {
      job.log(`Processing failed: ${error.message}`);
      throw error; // Rethrow to handle retries or errors
    }
  }
}

Progress Tracking and Error Handling

Implement progress tracking to keep users informed about the status of their documents. Use the Bull's progress feature to update the status of each job. Expose an API endpoint that allows users to query the status of a specific document. This can be done by using the job ID. Ensure that all the operations are properly wrapped in try-catch blocks to catch any exceptions. Log errors for debugging, but don't expose sensitive information. Implement retry mechanisms to handle transient errors, such as temporary network issues. Use logging and monitoring tools to track the application's performance and error rates. This helps in proactively identifying and addressing issues.

File Validation and Sanitization

Implement file validation to ensure the uploaded files meet the required criteria. Use file type validators to only accept the allowed file formats. Set the file size limits and reject any files that exceed those limits. Sanitize filenames to prevent any security vulnerabilities. Handle file errors gracefully. Return informative error messages to the users. This helps in guiding the users and providing them with information about the issues and how to fix them.

S3 or Local Storage Integration

Choose either S3 or local storage for file storage. We'll set up integration with S3 to store the uploaded files. This will involve configuring the AWS SDK and creating a bucket. Define the access permissions. Make sure that the uploaded files can be accessed by the service. Store the metadata information of files into the database. This includes file names, sizes, and any other relevant information. For local storage, create a directory to store the files and save the files in the directory. Create an API to access the files.

Testing, Documentation, and Deployment

Now that our core features are set up, we'll shift our focus to testing, API documentation, and deployment. These steps ensure that our application is robust, user-friendly, and ready for production.

Unit and Integration Tests

Implement unit tests for each component. This means testing individual functions and classes to ensure they function as expected. Write integration tests to verify that different components work together correctly. Use tools like Jest to run and manage your tests. Regularly run tests during the development process to catch issues early. This helps in maintaining code quality.

API Documentation with Swagger

Generate API documentation using Swagger to make it easier for developers to understand and interact with the API endpoints. Annotate your controllers and DTOs with Swagger decorators. Configure Swagger to generate an interactive API documentation. This makes it easier for others to understand and use your API.

import { ApiOperation, ApiResponse, ApiBody } from '@nestjs/swagger';

@Controller('documents')
export class DocumentProcessingController {
  @Post('upload')
  @ApiOperation({ summary: 'Upload a document for processing' })
  @ApiBody({
    schema: {
      type: 'object',
      properties: {
        file: {
          type: 'string',
          format: 'binary',
        },
      },
    },
  })
  @ApiResponse({ status: 201, description: 'Document uploaded and processing started.' })
  async uploadFile(@UploadedFile() file: Express.Multer.File) {
    // Implementation
  }
}

Deployment Strategies

Choose the appropriate deployment strategy for the service. You can use containerization with Docker and deploy to a cloud platform like AWS, Google Cloud, or Azure. Configure the environment variables for your application, such as database credentials and S3 bucket information. Automate the deployment process using CI/CD pipelines. This automates the build, test, and deployment of your application.

Conclusion: Your Document Processing Toolkit

Congratulations! You've built a powerful Document Processing Service with NestJS. You've learned about document upload, OCR processing, metadata extraction, and storage, while also getting hands-on experience with important tools like Bull, Tesseract.js, and TypeORM. Your new toolkit is ready to tackle various document-centric challenges. Keep learning, keep experimenting, and happy coding!