PDF《Data Streaming》

0 while True: header = sock.recv() print header while socket.getsockopt(zmq.RCVMORE) data = sock.recv() open('file_{}.raw'.format(counter),'wb') f.write(data) f.close() counter +=

1 ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ZMQ HWM (HighWaterMark) HWM PUSH/PULL PUB/SUB Delivery Scheme: Control Control ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence Splitter PUSH/PULL PUSH/PULL ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence Splitter PUSH/PULL PUSH/PULL Viewer(s) PUB/SUB ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence Splitter PUSH/PULL PUSH/PULL Viewer(s) PUB/SUB Analysis PUSH/PULL PUSH/PULL or PUB/SUB ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design - Advantages No delay in online analysis and viewing Re-use of components (e.g. file writing) Adding/exchange of components (e.g. detector, processing) Dynamic and automatic scaling (e.g. of processing) due to push/pull scheme Easy debugging through JSON header ... ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design - Disadvantages It is a distributed system with all its challenges ... ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Detectors GigaFROST Pixels: 2016*2016 /

12 bit Exposure: 2?s exposure

1255 Hz Data Rate: 61.21Gbit/s - 7.651 GByte/s Write Stream:

2 GByte/s Preview Stream:

5 Hz PCO Edge Pixels: 2560x2160 /

16 bit Speed:

100 Hz mit dem 286Mhz ADC Rolling Shutter Data Rate: 1GByte/s Write Stream: 1GByte/s Preview Stream ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Lessons Learned ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Lessons Learned Keep code of components simple (no all-in-one-components) Failure tolerant design of components/software Automatic restart of components ... Take special care when starting and closing a connection Setting ZMQ high watermark (on client and server) "right" is very important Central logging is a must Statistics for all components are needed Messages received Messages dropped ... ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Outlook ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Outlook - Improvements Improve tooling for Python, Java, … Improve (central) debugging and logging Work on the easy configuration and orchestration of the components/modules (black box concept ) … ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Interested ? ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Contact Us ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Questions ? ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Simon Ebner Paul Scherrer Institute WBGB/001

5232 Villigen PSI [email protected] Contact Slides ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Colors