Node.js Stream Module

Java FileNode.js Built-in Modules

Stream is an abstract interface in Node.js for handling streaming data. You can think of a stream as a water flow, where data flows from one place to another like water. This processing approach is especially suitable for large files or continuous data sources, because it does not need to load all data into memory at once.

Why Do We Need Stream?

  1. Memory efficiency: When processing large files, there is no need to load the entire file into memory at once
  2. Time efficiency: Data can be processed as it is received, without waiting for all data to be ready
  3. Composability: Multiple streams can be connected like pipes to form a data processing pipeline

Four Basic Types of Stream

1. Readable Stream

A readable stream is the source of data, from which data can be read. For example, reading content from a file or obtaining data from an HTTP request.

Example

const fs = require('fs');
const readableStream = fs.createReadStream('example.txt');

readableStream.on('data', (chunk) => {
  console.log(`Received ${chunk.length}bytes of data`);
});

readableStream.on('end', () => {
  console.log('No more data');
});

2. Writable Stream

A writable stream is the destination of data, into which data can be written. For example, writing to a file or sending an HTTP response.

Example

const fs = require('fs');
const writableStream = fs.createWriteStream('output.txt');

writableStream.write('First line of data\n');
writableStream.write('Second line of data\n');
writableStream.end('Last line of data');

3. Duplex Stream

A duplex stream is both readable and writable, such as a TCP socket.

Example

const { Duplex } = require('stream');

const myDuplex = new Duplex({
  write(chunk, encoding, callback) {
    console.log(chunk.toString());
    callback();
  },
 
  read(size) {
    this.push('Data read from the duplex stream');
    this.push(null); // Indicates the end of data
  }
});

4. Transform Stream

A transform stream is a special type of duplex stream that can modify or transform data during the reading and writing process. For example, compressing or encrypting data.

Example

const { Transform } = require('stream');

const upperCaseTr = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

process.stdin.pipe(upperCaseTr).pipe(process.stdout);

Common Methods of Stream

pipe() Method

pipe()The method is one of the most powerful features of Stream; it can connect multiple streams together to form a pipeline.

Example

const fs = require('fs');

// Create a readable stream and a writable stream
const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');

// Use pipe to connect the readable stream and the writable stream
readStream.pipe(writeStream);

writeStream.on('finish', () => {
  console.log('Data write complete');
});

Event Listening

Stream is an instance of EventEmitter. Various situations can be handled by listening to events:

  • data- Triggered when data is available to read
  • end- Triggered when there is no more data to read
  • error- Triggered when an error occurs
  • finish- Triggered when all data has been flushed to the underlying system

Practical Application Examples

Example 1: File Copy

Example

const fs = require('fs');

function copyFile(source, target, cb) {
  // Create read and write streams
  const rd = fs.createReadStream(source);
  const wr = fs.createWriteStream(target);

  // Handle errors
  rd.on('error', err => cb(err));
  wr.on('error', err => cb(err));
 
  // Completion callback
  wr.on('finish', cb);
 
  // Start copying
  rd.pipe(wr);
}

// Usage
copyFile('source.txt', 'target.txt', err => {
  if (err) console.error(err);
  else console.log('Copy complete');
});

Example 2: HTTP Server Compressing Files

Example

const fs = require('fs');
const http = require('http');
const zlib = require('zlib');

http.createServer((req, res) => {
  // Create a readable stream
  const readStream = fs.createReadStream('./largeFile.txt');
 
  // Set response headers
  res.writeHead(200, {
    'Content-Type': 'text/plain',
    'Content-Encoding': 'gzip'
  });
 
  // Pipeline chain: read -> compress -> respond
  readStream.pipe(zlib.createGzip()).pipe(res);
}).listen(3000);

Stream Best Practices

  1. Error handling: Always add error handling listeners to streams
  2. Memory management: For large files, use streams instead of reading everything at once
  3. Pipeline chain: Use the pipe() method appropriately to combine multiple streams
  4. Backpressure handling: When the write speed cannot keep up with the read speed, backpressure needs to be handled appropriately

Error Handling Example

Example

const fs = require('fs');

const readStream = fs.createReadStream('not-exist-file.txt');

readStream.on('error', (err) => {
  console.error('Error occurred:', err.message);
});

By mastering the Node.js Stream module, you can efficiently handle various I/O operations. Especially when processing large files or real-time data, streams will become your powerful tool.

Java FileNode.js Built-in Modules

Other Extensions