Update setup doc with DLQ details

Author: billsedison

KAFKA_SETUP.md

@@ -106,6 +106,36 @@ docker exec -it kafka kafka-console-consumer --topic avscan.action.scan --from-b
 2. Register the handler in the KafkaModule providers array
 3. The handler will automatically be registered and start consuming messages
 
+### Dead Letter Queue (DLQ) Support
+
+The application includes a robust Dead Letter Queue implementation for handling message processing failures:
+
+1. **Configuration**:
+   ```
+   # DLQ Configuration in .env
+   KAFKA_DLQ_ENABLED=true
+   KAFKA_DLQ_TOPIC_SUFFIX=.dlq
+   KAFKA_DLQ_MAX_RETRIES=3
+   ```
+
+2. **Retry Mechanism**:
+   - Failed messages are automatically retried up to the configured maximum number of retries
+   - Retry count is tracked per message using a unique key based on topic, partition, and offset
+   - Exponential backoff is applied between retries
+
+3. **DLQ Processing**:
+   - After exhausting retries, messages are sent to a DLQ topic (original topic name + configured suffix)
+   - DLQ messages include:
+     - Original message content
+     - Error information
+     - Original topic, partition, and offset
+     - Timestamp of failure
+     - Original message headers
+
+4. **Monitoring DLQ**:
+   - Use Kafka UI to monitor DLQ topics (they follow the pattern `<original-topic>.dlq`)
+   - Check application logs for messages with "Message sent to DLQ" or "Failed to send message to DLQ"
+
 ### Monitoring and Debugging
 
 - **Application Logs**: Check console output for Kafka connection status and message processing
@@ -121,6 +151,10 @@ All Kafka-related environment variables are documented in `.env.sample`:
 - `KAFKA_GROUP_ID`: Consumer group ID
 - `KAFKA_SSL_ENABLED`: Enable SSL encryption
 - Connection timeouts and retry configurations
+- **DLQ Configuration**:
+  - `KAFKA_DLQ_ENABLED`: Enable/disable the Dead Letter Queue feature
+  - `KAFKA_DLQ_TOPIC_SUFFIX`: Suffix to append to original topic name for DLQ topics
+  - `KAFKA_DLQ_MAX_RETRIES`: Maximum number of retries before sending to DLQ
 
 ## Troubleshooting
 
@@ -129,6 +163,7 @@ All Kafka-related environment variables are documented in `.env.sample`:
 1. **Connection Refused**: Ensure Kafka is running with `docker compose -f docker-compose.kafka.yml ps`
 2. **Topic Not Found**: Topics are auto-created by default, or create manually using Kafka UI
 3. **Consumer Group Issues**: Check consumer group status in Kafka UI under "Consumers"
+4. **DLQ Topics Missing**: DLQ topics are created automatically when the first message is sent to them
 
 ### Cleanup
 
@@ -145,5 +180,9 @@ docker compose -f docker-compose.kafka.yml down -v
 - Configure SSL/TLS and SASL authentication for production environments
 - Set appropriate retention policies for topics
 - Monitor consumer lag and processing metrics
-- Configure dead letter queues for failed messages
-- Set up proper alerting for consumer failures
+- Ensure DLQ topics have appropriate retention policies (longer than source topics)
+- Set up alerts for:
+  - Messages in DLQ topics
+  - High retry rates
+  - Consumer failures
+- Implement a process for reviewing and potentially reprocessing DLQ messages