Skip to main content

Elasticsearch

Elasticsearch plays a central role in the tSM platform, enabling denormalized storage, rich querying capabilities, and historical/audit data retention. This chapter explains how Elasticsearch integrates with the tSM microservice ecosystem, its role in CQRS and event sourcing patterns, and how it leverages our custom tQL (tSM Query Language).

1. Why Elasticsearch?​

In a microservice architecture, data is often split across multiple services and databases. While this approach is highly modular and scalable, it can make it difficult to query and report on data that spans multiple domains. Elasticsearch addresses this challenge by providing:

  • Denormalization: tSM consolidates key entities—such as Customer, Account, Ticket, Order, Inventory Item, and more—into denormalized Elasticsearch documents.
  • High-Performance Search: Full-text search capabilities and near real-time updates via Kafka ensure that current data is always available for queries.
  • Aggregations and Analytics: Elasticsearch’s aggregation engine supports operational reporting, dashboards, and analytics use cases.
  • Flexible Data Aging: Using Index Lifecycle Management (ILM), you can set data retention policies to automatically archive or delete older data (e.g., historical logs and process instances).

2. Denormalized Data Model​

Each major entity in tSM is represented as an Elasticsearch “document” that combines related fields into one place. For example, a Ticket document might include:

  • Ticket metadata (ID, status, assigned user)
  • Customer details (name, contact info)
  • Related comments or notes (including AI-derived sentiment tags)
  • References to other entities like Orders or Inventory Items

By denormalizing these attributes, tSM can quickly retrieve or search across all related data with minimal joins or cross-service lookups.

How Data Gets to Elasticsearch​

  1. Change Data Capture (CDC): Whenever a microservice database (e.g., PostgreSQL, Oracle, SQL Server) is updated, a CDC mechanism sends event records to Apache Kafka.
  2. Kafka Consumer: A dedicated consumer (part of a tSM microservice or a specialized indexing service) reads these CDC events.
  3. Indexing: The consumer transforms and merges the data into the relevant Elasticsearch document. Fields from different microservices can be combined here.

As a result, Elasticsearch always contains the latest state across multiple entities in a denormalized form.

3. tQL (tSM Query Language)​

tQL is a SQL-like language designed to query and analyze data stored in Elasticsearch. It powers many of tSM’s features:

  • Listings (e.g., Ticket listings, Customer listings)
  • Reporting & Dashboards (aggregations and charts)
  • Dedicated tQL Console (for query development and debugging)
  • API Queries (REST endpoints that accept tQL syntax)
  • Access Rules & SLAs (filters based on tQL expressions)

tQL vs. SQL​

While tQL resembles Elasticsearch SQL or OpenSearch SQL, it includes specialized features:

  • Enhanced Nested Object Support: Ideal for denormalized documents containing arrays of objects.
  • Column Definitions: tSM consults metadata about fields (types, custom converters) before generating the Elasticsearch query.
  • Post-Processing: tQL can run calculations after Elasticsearch returns data. For example, converting priorityId to a human-readable priority name.

For more detailed information, please see the tQL Documentation.

4. CQRS Pattern​

Elasticsearch in tSM implements the “Query” side of a CQRS (Command Query Responsibility Segregation) pattern:

  • Command side: Transactional microservices update their local relational or graph databases (e.g., PostgreSQL, Oracle, Neo4j).
  • Query side: Data changes are broadcast via Kafka and indexed in Elasticsearch for fast, flexible queries.

This division allows microservices to optimize their internal data stores for write operations without compromising the speed and complexity of read queries.

5. Event Sourcing Flavors​

While tSM does not use full event sourcing for every domain entity, it does adopt certain event sourcing concepts:

  1. Events: Changes (e.g., a new comment on a Ticket, an Order status update) flow through Kafka.
  2. Consumers: Microservices or specialized listeners subscribe to these event topics.
  3. Projections: Interested services update Elasticsearch or perform additional logic.

Example:

  • A custom AI microservice can listen for “comment created” events, determine the sentiment using NLP, and then update the corresponding Ticket document in Elasticsearch.
  • This new attribute (e.g., commentSentiment) becomes instantly available for queries and reporting in tSM.

6. Historical and Audit Data​

tSM uses Elasticsearch as the primary storage for:

  • Audit Logs: Track changes, actions, or workflow steps.
  • Process Instances History: Store the entire lifecycle of workflow executions.
  • Integration Logs & Notifications: Keep a searchable record of external integrations and message exchanges.

Since these logs can grow large over time, Elasticsearch ILM (Index Lifecycle Management) is leveraged to automatically roll over older indices and eventually archive or purge them according to organizational retention policies.

7. Putting It All Together​

By unifying data in Elasticsearch, tSM achieves:

  • Single View of Entities: All relevant entity data is quickly retrievable from one document.
  • Powerful Queries & Analytics: Through tQL, users and microservices can run complex searches, aggregations, and charts without straining transactional databases.
  • Extensibility: New microservices can plug into the event stream, update Elasticsearch with custom fields, and leverage them in queries—without changing the core platform.
  • Scalability: Elasticsearch’s distributed architecture and Kafka’s high-throughput messaging ensure that tSM can handle large volumes of data and events.

Next Steps​

  • Read More About tQL: tQL Documentation
  • Deployment Guide: Learn how to scale Elasticsearch in a containerized environment using Docker/Kubernetes.
  • Monitoring: See Monitoring & Observability for integrating Elasticsearch with Kibana, Prometheus, and Grafana.

This concludes our overview of Elasticsearch in the tSM Architecture. By employing denormalization, event-driven updates, and tQL-based queries, tSM ensures a robust, flexible, and performance-oriented data platform for your business needs.