AWS Kinesis Notes

Basic Examples

Simple cloudformation example Don’t use EventSourceMapping. Just add event trigger with lambda function definition.

Simple read

Simple write example:

const AWS = require('aws-sdk');

exports.handler = function(event, context) { 
  
  try {
    const kinesis = new AWS.Kinesis({
      apiVersion: '2013-12-02',
      region: process.env.region
    });
    const kinesisStreamName = "KinesisStream1";  
    let date_ob = new Date();

    let params = {
      Data: date_ob.toString(),
      PartitionKey: '1',
      StreamName: kinesisStreamName
    };
    
    
    kinesis.putRecord(params, function(err, data) {
      if (err) {
        console.log('Kinesis Record Error: ' + JSON.stringify(err, err.stack));
      } else {
        console.log('Kinesis Record added: ');
      }
      return;
    });

  }catch(error){
        console.log("Error: " + error.message);
        return;
  }
  
}

Monitor hot shard with Cloudwatch

Starting points:

AT_TIMESTAMP

— from specific time stamp

TRIM_HORIZON

— all the available messages in Kinesis stream from the beginning (same as earliest in Kafka)

LATEST

Lambda retry behavior

If the function receives the records but returns an error, Lambda retries until the records in the batch expire, exceed the maximum age, or reach the configured retry quota. For function errors, you can also configure the event source mapping to split a failed batch into two batches. Retrying with smaller batches isolates bad records and works around timeout issues. Splitting a batch does not count towards the retry quota.