pratiks
12/6/2017 - 3:18 PM

Kafka-Notes-Lozano

serialize messages to kafka - use middleware 



notes: 

serde -  serialization library

sync 
 -- producer.produce(ctx, pm) -> sends to kafka 
 -- sent to chat.message  
 -- consumers: messageHydration, appIntWorker, hcopenchats-workers, hc-filter-messages-worker

-  hc-filter-messages-worker consumes chat.message 
    - checks if its a direct or a group 
    - categorizes direct / group
    - has a producer and a consumer 
    if direct -> send notificationWorker -> notificiationWorker 
    if group -> send to another worker for further processing. 

    - order log of events - partitions ( do not confuse this with replica partition)
    - configure # of consumers for a topic 
    - 1 2to 
    - consumer groups scale up and down on their own  on the workers 


    - commiting the offset means you read the message 
    - each consumer knows where it reads from the name of topic, offset