Systems Integration using Apache Camel 1 Mgr. Ivo Bek Senior Product Manager April 2023 Starting a new business 2 Which products / services would you choose today? What next? 3 Respond to new needs and expectations ▸ AI, assistants, bots, … ▸ Nanobots, robotics ▸ Quantum computing ▸ AR / VR ▸ Brain computer interfaces ▸ Space network - laser comms, interplanetary internet, ▸ Synthetic biology ▸ Bioengineering ▸ …. 4 HOW? Existing System New capabilities New architectures and approaches New services New experiences When would you rewrite parts of the system? 5 When would you rewrite parts of the system? 6 ▸ Products and services are out of support or close to it ▸ No or very few people have skillsets for those products and services ▸ Expensive to run ▸ Bad UX ▸ Low performance ▸ … When would you keep parts of the system? 7 When would you keep parts of the system? 8 ▸ It works ▸ Critical part of the system ▸ Products and services are still supported ▸ The technology is still widely used ▸ Replacement would cause expensive downtime ▸ Big risk of failing ▸ Rewriting would mean huge investments To run and innovate our business 9 We integrate ▸ Balance ▸ Low risk & High gain ▸ Do it smart using ･ Standard communication protocols ･ Standard data formats ･ Proofed integration patterns ▸ Stay agile and choose the best architectures for any given problem ･ Microservices, Service mesh ･ Event-driven ･ Hierarchical ･ Hexagonal ･ Gateway ･ Serverless ･ … … because the world is not getting any simpler Evolution of Integration 10 Point to Point Direct connection between systems, application both internally and with external services Enterprise Service Bus Placing a centralized bus that integrate between loosely coupled services. Microservices Fine grained distributed services, allowing faster turnover rate, more agile and flexible deployment model. Serverless Scale down to zero. Optimize Resource Usage. Avoid random, arbitrary workload prediction Architecture ENTERPRISE INTEGRATION What are some of the common integrations today? 11 Expose legacy SOAP web services using REST API SOAP WebService REST API Translate WSDL E.g. OpenAPI What are some of the common integrations today? 12 Bridge messages between different Broker technologies ▸ Plenty of vendors and products ▸ Various message types and protocols ･ JMS ･ AMQP ･ MQTT ･ Kafka ･ CloudEvent FACTORY EDGE SYSTEM 13 Example machine condition monitoring based on sensor data in an industrial setting, using AI/ML CloudNearedgeFaredge Camel as messaging bridge between MQTT and Kafka events INTEGRATION PATTERNS 14 ▸ Data handling ▸ Data model translation ▸ Data synchronisation ▸ Publish subscribe ▸ Routing ▸ Content based routing ▸ Data enrichment ▸ Protocol and transport translation ▸ Large object management ▸ Correlation ▸ Assured delivery ▸ Canonical data model High-level vision of integration System implementation Enterprise Integration Patterns ▸ Relationships/cross referencing ▸ Once only delivery ▸ Scatter gather ▸ Aggregation ▸ Store and forward ▸ Flow control/throttling ▸ Service exposure ▸ Failed event management ▸ Idempotence resolution ▸ Retry ▸ Health check ▸ Prioritization ▸ Optimistic/pessimistic locking ▸ Event sequencing ▸ Verb conversion ▸ Existence checking ▸ Compensation ▸ Batch processing (various) ▸ Command controller (various) ▸ Error handling (various) ▸ Composition (various) ▸ Orchestration (various) 15 ? That could connect to any almost any system With support for known integration patterns That can work on and off the cloud Integration defined in a simple language. Such as XML, YAML and Java from(“kafka:topic”) .to(“grpc:endpoint”) This is Apache Camel Apache Camel APPLICATION FOUNDATIONS Apache Camel 16 Basics Kamelet Concepts 17 Kamelets are use case driven, they are Camel recipes encapsulating a well defined purpose KMLT Kubernetes Resource Use case story... … implemented as a Kamelet Types of Kamelets 18 Forward data from the platform to external systems. ActionsSinks EIPs, transformation, etc Sources Bring data from external systems to the platform Camel with Kamelets 19 Use Kamelets where you need, as sources, sinks, or mid-flow actions Source SinkAction KMLT Incoming Data Outgoing Data KMLT KMLT Camel-K route ... ... more Camel-K code ENTERPRISE INTEGRATION bind Kamelets to form a running integration unit Kafka client Kafka app Kafka app Kafka client Kamelet Bindings EXAMPLE CLOUD-NATIVE INTEGRATION JOURNEY 21 Integration logic Prototype Test Debug Integration image Run with hot-reload Local dev env Cloud dev env Automatic dependency resolution for integration components Serverless Service binding Event-based autoscaling Metrics Tracing Integration service > kamel run kafka-http.yaml --dev - from: uri: kafka:example-topic steps: - log: message: ${body} logging-level: INFO - to: uri: ${TARGET_URL} kafka-http.yam l EXAMPLE CLOUD-NATIVE INTEGRATION JOURNEY 22 Integration logic Prototype Test Debug Integration image Run with hot-reload Local dev env Cloud dev env Automatic dependency resolution for integration components Serverless Service binding Event-based autoscaling Metrics Tracing Production env Promote Integration service Integration service Camel in your projects 23 ▸ Run integration service(s) in a local environment ▸ No programming but writing integration DSL Camel route DSL Camel route DSL Routing, Transformation, Processing, Filtering, … Camel route DSL Camel route example 24 HTTP -> Log Hello ${body} from Camel Summary 25 ▸ Technologies evolve and businesses need to adapt effectively through integration in order to innovate and stay competitive ▸ Apply well defined practices and patterns to integrate services and systems ▸ Use standard communication protocols, messaging and data formats ▸ Stay agile to combine and use mix of architectures ▸ Apache Camel is the most popular integration framework, open source and free, bookmark it CAMEL LAB 26 Installation 27 ▸ JBang - curl -Ls https://sh.jbang.dev | bash -s - app setup” ▸ Camel JBang - jbang app install camel@apache/camel ▸ Run and Verify $ camel version Camel JBang version: 3.Y.Z Create your first integration service 28 ▸ Platform HTTP /hello -> Log “Hello ${body} from Camel” $ camel init hello.xml $ camel run hello.xml $ curl -X POST http://0.0.0.0:8080/hello -H "Content-Type: text/plain" -d "Ivo" Hello Ivo from Camel Read json property 29 ▸ Platform HTTP /hijson -> Log “Hi ${body[name]} from Camel” application/json $ curl -X POST http://0.0.0.0:8080/hijson -H "Content-Type: application/json" -d "{\"name\":\"Ivo\"}" Hi json Ivo from Camel Prepare DB 30 ▸ SQLite $ sqlite3 data.db CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, UNIQUE("name") ); camel.beans.myDataSource = #class:org.sqlite.SQLiteDataSource camel.beans.myDataSource.url = jdbc:sqlite:/home/ibek/git/pv207/data.db camel.component.sql.dataSource = #bean:myDataSource application.properties Integrate with DB 31 ▸ SQLite ▸ Platform HTTP /users POST -> SQL Insert User ${body[name]} added successfully $ curl -X POST http://0.0.0.0:8080/users -H "Content-Type: application/json" -d "{\"name\":\"Ivo\"}" User Ivo added successfully $ camel run integration.xml application.properties --deps=org.xerial:sqlite-jdbc:3.41.2.1 --dev Integrate with DB 32 ▸ SQLite ▸ Platform HTTP /users/{user} GET -> SQL Select $ curl http://0.0.0.0:8080/users/Ivo {"id":1,"name":"Ivo"} Transform json 33 ▸ JQ - remove id property del(.id) $ curl http://0.0.0.0:8080/users/Ivo {"name":"Ivo"} Direct routes 34 The Direct component provides direct, synchronous invocation of any consumers when a producer sends a message exchange. This endpoint can be used to connect existing routes in the same camel context. Filter EIP 35 ${body[name]} !regex '.*-bot' $ curl -X POST http://0.0.0.0:8080/filteredusers -H "Content-Type: application/json" -d "{\"name\":\"John-bot\"}" Choice EIP 36 ${body[total]} < 100 $ curl -X POST http://0.0.0.0:8080/order -H "Content-Type: application/json" -d "{\"total\":50}" Learn more about Camel 37 ▸ https://camel.apache.org ▸ More integration pattern examples at: https://camel.apache.org/components/3.20.x/eips/enterprise-integration-patterns.html ▸ More components to integrate with: https://camel.apache.org/components/3.20.x/ linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Red Hat is the world’s leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you 38